Patterico's Pontifications

1/27/2013

We Don’t Know As Much As We Think We Do About the Economy, Part 3: Don’t Ever Say “A Study Shows” To Prove Anything

Filed under: General — Patterico @ 2:02 pm



I have been discussing what I learned in a podcast in which Jim Manzi cautions people to be very skeptical about the authority of conclusions reached by social scientists, including economists. The point he consistently makes is that, to have reliability, any phenomenon should be repeatable in different situations in different contexts, allowing people to make consistent and reliable non-obvious predictions. One important corollary is not to make too much of a single study — no matter how interesting the result might be, or how much it seems to confirm your pre-existing biases.

We all do this. Here’s a fun real-world example showing why it’s a bad idea.

Economic theory tends to hold that greater choice leads to greater demand and consumption — thus, if you want to sell more, offer more choices to your consumers. But Manzi tells the story of researchers who set out to test this. They ran an experiment in which they set up a table at a grocery store on two successive Saturdays. At the table, they had a selection of jams and jellies. One Saturday they had a selection of six different jams and jellies, and the other Saturday they had 24 varieties. On each day, they asked shoppers to taste their wares, and if they liked them, the shopper would be given a dollar coupon to redeem at the checkout counter to purchase one of the jams or jellies.

Conventional economic theory would hold that the day where the greater choice was available, a higher percentage of coupons would be redeemed. But the researchers found something counterintuitive and interesting. On the day where they had six different jams or jellies for purchase, fully 30% of the shoppers used a coupon to buy jams or jellies. On the day when they had 24 different varieties of jams or jellies, only 3% of shoppers redeemed the coupon.

Fascinating, huh? Reducing choice actually increased sales. You can easily see journalists writing an article that wows the public. Call the Freakonomics guys. Can’t you envision a section of a chapter talking about this surprising result, and discussing the likely reasons for it? Perhaps the shoppers were paralyzed by indecision when presented with so many choices. Valuable information, certainly, for any marketer to know.

Valuable — and almost certainly wrong.

Manzi says it was a mistake to draw sweeping conclusions from this single experiment. After all, look at how extreme the results are.

Can it really be true that all you have to do to increase sales by a factor of 10 is to remove 75% of your supply from your shelves? If this is the case, Manzi says, retailers everywhere are leaving “suitcases of money on the ground.” That is a HUGE effect — and frankly, so remarkable that it should raise a red flag.

The effect was so remarkable, in fact, that people have tried to replicate this experiment using other products, in other contexts, to see if the results can be repeated. And they can’t be. Manzi says that the experiments produuce a wide range of distributions — and that the general trend is that greater choice leads to greater demand.

Remember this the next time a single social science study comes to a conclusion you like. It’s tempting to cite such a study — we’re almost all guilty of this. But I am personally going to try to be aware of this in the future. Unless a phenomenon is proven through repeated observation in different contexts, allowing people to make repeatable, reliable, non-obvious predictions, the result of a social science study should always be viewed with immense skepticism.

56 Responses to “We Don’t Know As Much As We Think We Do About the Economy, Part 3: Don’t Ever Say “A Study Shows” To Prove Anything”

  1. Ding.

    Patterico (8b3905)

  2. after tasting 24 jams and jellies I’d be pretty much over jams and jellies for awhile too

    plus if you buy jams and jellies you know what you’re doing, yes?

    be honest

    you’re secretly plotting to eat carbs with them, cause that’s the only thing jams and jellies are good for… it’s their whole raison d’être!

    so when this happens I just think of the wisdom I found one day on my friend J’s pinterest page:

    Nothing tastes as good as skinny feels.

    happyfeet (ce327d)

  3. This is an absurd straw man. The silly argument that Manzi wants props for demolishing was touted in a book by a sociologist, Barry Schwartz (The Paradox of Choice). I don’t think that more than a tiny number of economists looking for heterodox things to write put any stock in it.

    There are tons of “studies” out there. Some are very carefully done, others are obvious crap. A dozen poorly done studies aren’t as persuasive to me as a single well-executed one with credible data.

    A pretty good rule of thumb is that if the “study” is being put out by a “policy institute” of any political stripe, you should be wary.

    Chip S. (07cd30)

  4. Patterico–any chance that you might consider repositioning the part 1, 2 and 3 to the beginning of the thread name instead at the middle or end? It’s getting kind of confusing and cluttered over on the “recent comments” side of the screen.

    elissa (f1f945)

  5. Most social science studies begin with a premise which is soon validated by those selfsame studies. Call it the Dirty Old Man Rule: social scientists rarely falsify studies in which they have an ideological, emotional, or financial interest.

    Dirty Old Man (0c7e45)

  6. This is an absurd straw man. The silly argument that Manzi wants props for demolishing was touted in a book by a sociologist, Barry Schwartz (The Paradox of Choice). I don’t think that more than a tiny number of economists looking for heterodox things to write put any stock in it.

    He doesn’t want props for demolishing it. He is using it as an example of how ONE study in a social science discipline cannot possibly be considered reliable because there are so many potential variables that can make the results questionable.

    It’s a good point that you seem to overlook when you say you’d be willing to put stock in a single study if it had “reliable data.”

    Patterico (879b60)

  7. Elissa,

    Hard to do on a phone, sorry.

    Patterico (879b60)

  8. What you should get out of this, is that observational studies can’t be trusted. This is true with food studies where clinical trials often debunk observational studies. The jelly study falls under observational study. Here’s a good video on what we should be looking out for on food observational studies. I believe you can apply it to other observational studies.

    http://www.youtube.com/watch?v=y1RXvBveht0

    Tanny O'Haley (12193c)

  9. It’s a good point that you seem to overlook when you say you’d be willing to put stock in a single study if it had “reliable data.”

    If the point is that more studies are better than fewer studies, it’s not so much a point as a statement of the blindingly obvious.

    If your point is that you can’t tell what data sources are more reliable than others, then you probably shouldn’t pay attention to any studies at all.

    Chip S. (07cd30)

  10. If the point is that more studies are better than fewer studies, it’s not so much a point as a statement of the blindingly obvious.

    I believe I made the point quite clearly. You seem to be having a difficult time restating it accurately. Perhaps emotion is getting in the way?

    Patterico (1a103d)

  11. Studies show and many people say that scientific studies, aren’t.

    Jcw46 (eda37d)

  12. i love that thing where you take the jalapeno jelly and just pour it over a brick of cream cheese and serve with a little cheese knife and crackers

    nobody ever does that here in california

    or maybe they just never invite me

    happyfeet (ce327d)

  13. “scientists are freaking liars”

    Tanny’s link @ #8 is worth the time

    happyfeet (ce327d)

  14. if
    i only had an ocean

    Pittsburg Pa

    EVERUY BODY GONE SU..Opp
    serfin

    usa

    pdbuttons (949c1e)

  15. hf,

    Have you ever had cream cheese with wasabi paste, sesame seeds and soy sauce? It’s also good with green onion, cilantro, or minced fresh jalapeno.

    DRJ (a83b8b)

  16. that sounds really really good and much more suited to the california palate – do you mix it all together or just make a layer on top of the cream cheese?

    happyfeet (ce327d)

  17. Wasabi improves many things. Especially peas.

    JD (b63a52)

  18. I’m not eating my peas. I’m protesting.

    daleyrocks (bf33e9)

  19. happyfeet,

    I haven’t made it in a long time but I used to love it. Cut the cream cheese block in half (either direction, depending on the size of the plate you’re going to serve it on), put the wasabi paste in the middle, and then put the cream cheese blocks back together. Roll in toasted sesame seeds, and pour soy sauce over the top, garnishing with chopped green onion, cilantro, and/or minced fresh jalapeno. I like to use Triscuits but you can also serve it with rice crackers.

    DRJ (a83b8b)

  20. that’s something I’m definitely trying in the very near future I’ll let you know how it goes

    happyfeet (ce327d)

  21. “Conventional economic theory would hold that the day where the greater choice was available, a higher percentage of coupons would be redeemed. But the researchers found something counterintuitive and interesting. On the day where they had six different jams or jellies for purchase, fully 30% of the shoppers used a coupon to buy jams or jellies. On the day when they had 24 different varieties of jams or jellies, only 3% of shoppers redeemed the coupon.”

    This is well known in medical decision theory. The New England Journal published a study about 20 years ago supporting this. If you offer a patient with arthritis a choice of several drugs for arthritis, they are less likely to fill the prescription than if only one or two are offered.

    There is a lot of decision theory research on this topic. If a student is on his way to the library to study and is met by another student who invites him to a party, the first student is less likely to do either and more likely to just go back to his dorm room.

    It’s why car salesmen show the customer only one car at a time. Lots of research on this.

    Mike K (dc6ffe)

  22. I wonder what would happen if you let people choose between a fascist chicago economy-raper and a bland convictionless establishmentarian technocrat

    happyfeet (ce327d)

  23. What the excitement of the Whitman campaign, didn’t give you a tingle, have a cupcake.

    narciso (3fec35)

  24. “I wonder what would happen if you let people choose between a fascist chicago economy-raper and a bland convictionless establishmentarian technocrat”

    Mr. Feets – Tough to say, but if u play ur cards right, wif a greasy azzed fascist chicago economy raper at least u got the chance of some graft coming right bak atchoo.

    daleyrocks (bf33e9)

  25. wheee it’s like chutes n ladders wif bonus fascisms!

    i wonder if i can get this on my phone

    happyfeet (ce327d)

  26. I wonder what a successful, organized society does when it gets a chance to elect representatives that cater to base human desires while forgetting that humans kind of like to achieve?

    I guess we will find out. As if we haven’t already.

    Ag80 (b2c81f)

  27. I think they called it the Roman Empire, it didn’t end well.

    narciso (3fec35)

  28. The case that free markets lead to the most efficient allocation of resources is completely dependent on the idea that consumers can correctly judge what is in their own self-interest. Because the welfare that is maximized in a free market is determined by that distribution of resources that maximize consumer consumption over all consumers. Arguments that question an individual’s capacity to make consumption decisions are nothing more than a backdoor into tyranny. The free market presumption about the individual is simply that he will pursue his own happiness as he sees fit. That fact that others might question his choice should be of no consequence. But this flies in the face of our therapeutic liberalism. Of course the bright people who went to Harvard can do a better job of choosing for the masses. That’s the whole point!

    Sadly, a lot of this is based on a consensus that the social sciences have much to tell about how we should govern, or more aptly be governed. Most social science “experiments” are not worth the time spent by the subjects let alone the researchers who devote entire careers to these charades IMHO. About the best thing that can be said of them is that they inspired the opening scene in Ghost Busters, and for that they will forever be of redeeming social value.

    However, there is a new kind of experiment that utilizes advanced technology, extensions of the now-familiar MRI are one example, that can begin to sort out how we actually process information. Leonard Mlodinow’s recent book, Subliminal: How your unconscious mind rules your behavior surveys these types of experiments and the results are fascinating. The first few chapters deal with the new technology and one of the startling results is that certain kinds of information will trigger the subconscious to change the path of information processing within the brain. His example is wine tasting using fMRI (Functional MRI) and the discovery is that telling a subject that one wine is very expensive, while the other is of only modest price, will change the parts of the brain that are active while tasting the wine samples, with the result that the expensive wine always tastes better … even when both are from the same bottle. Mlodinow also has a number of results relating to memory that are probably already familiar to all the lawyers and prosecutors who seem to populate this blog. But I found these results to be more subjective, less convincing, and tending to the old school of social science. Indeed Mlodinow strays far from his basic theme when makes the argument that global warming skeptics are suffering from some form of irrationality since there is a consensus that AGW is true.

    The question of the week is, of course, What difference does it make now? Before the federal government inserted itself into everyone’s life, it was quite respectable to have an eccentric uncle who squandered his inheritance on expensive wine that he thought exquisite, and ended up living with his older sister as the livein gardener into their dotage. But this is no longer tolerable. Foolish people are costing the government $billions and something must be done about it! I am always amused by doctors who complain about the irrational choices their patients make, and the apparent need for wiser people to make their choices for them. Presumably the clever overseers called for in this view of humanity will do what’s best for their clients, but history is not reassuring. Eugenics comes to mind. At any rate, my feeling is that even if we all make irrational choices, it is far better for us to have the freedom to make these choices than not. Mainly because limiting everyone’s choices to those setforth on some menu prepared by the federal government will kill off innovation, and with that, all hope for the future. I will even support Mlodinow’s right to irrationally conform to some alleged consensus even though he is amply trained to examine the scientific arguments and data, and then to judge for himself. He likes to get along, and that’s how he’s built. He will make his contributions in his own way.

    The Declaration of Independence said that the revolution was necessary to allow Americans to pursue their happiness. This was not a idle choice of words. Happiness is perceived by the individual, and it is different for us all. When each of us has the liberty to pursue our own happiness, then our country will prosper.

    bobathome (c0c2b5)

  29. Did they sell less jelly or just redeem less coupons?
    Did they sell more peanut butter than usual?

    Just kidding around. But I think it comes down to common sense. We all want a variety of choices… but we want those choices to come to us at a pace we can process easily.

    I’m just about smart enough and have the palate for five choices of jams and jelly.
    strawberry, peach, razzberry, blackberry, and blueberry. Tell me they all come seedless too and tell me all varieties are also offered in a sugarfree product. Tell me they’ve got peach, apricot, orange marmalade, etc and let me try them if I ask, But don’t put them all out on the same frickin table at once because I don’t have the time or palate cleansing ability to try them all. I’ll get exasperated within the process and just give up on you until you learn how to market multiple choices in a way that doesn’t overwhelm my palate and/or my patience

    steveg (831214)

  30. Interested to know if using the same test, condom coupon redemption dropped too.

    Anyway. It is all academic and best left to the central planners up in the politburo.

    steveg (831214)

  31. Compassionate conservatism in 2000 killed this country.
    They were in control and Bush never vetoed anything. Republicans and democrats suck and should be hung.

    mg (31009b)

  32. Ah, foodie recipes! Something I can actually use.

    Kiddo is really making me step up my game, all for nought. She’s playing me for the chump I be.

    gary gulrud (dd7d4e)

  33. “One Saturday they had a selection of six different jams and jellies, and the other Saturday they had 24 varieties. On each day, they asked shoppers to taste their wares, and if they liked them, the shopper would be given a dollar coupon to redeem at the checkout counter to purchase one of the jams or jellies”

    There is a time cost in testing and being rewarded for my efforts. I am more likely to spend time on a short survey for reward than sticking around and debating the merits of 24 different varieties. I have a family time schedule to keep, or places to be.. or things I’d rather be spending my time on. How much do you value your time in exchange for a $1 reward?

    tbflowers (11c057)

  34. This AM the 10-year Treasury yield has already hit 2.00% whereas was as low as 1.58% two months ago.

    Just in time for Feb. 15 to Mar. 15 rollover of $0.5 Trillion.

    gary gulrud (dd7d4e)

  35. Believe it or not, Spain will likely recover before we do.

    http://www.zerohedge.com/news/2013-01-28/spanish-housing-market-about-bottom

    We’re doing the tried and true Japanese model.

    gary gulrud (dd7d4e)

  36. 29. Comment by bobathome (c0c2b5) — 1/27/2013 @ 10:09 pm

    Stick around, we’re in need of a comprehensive vision.

    gary gulrud (dd7d4e)

  37. It is striking, isn’t it gary, who they think ignoring the lessons of the past, will work out, then again, Fallows wouldn’t be employed otherwise.

    narciso (3fec35)

  38. A wide variety of choices produces increased anxiety over trying to identify the optimum selection. Increased anxiety leads to delay and second guessing.

    Watch a woman trying to make a selection from an extensive restaurant menu.

    More choices equals less action.

    ropelight (59f33e)

  39. So, the lesson is we should have less choices, I don’t understand the point of the study, or maybe I do.

    narciso (3fec35)

  40. Pat

    Thanks for all these fine posts and the conversations these generated – I feel they will make an impact

    EPWJ (d84fb0)

  41. It was alot of work I hope people appreciate the time you take

    EPWJ (d84fb0)

  42. You have to take ZeroHedge with a huge handful of salt, but this piece has some important points

    SPQR (768505)

  43. ==Watch a woman trying to make a selection from an extensive restaurant menu. More choices equals less action.==

    uuh, ropelight-

    Are you not familiar with the Laboutin, Jimmy Choo element of this community and why we are called that and feared?

    elissa (ea9191)

  44. Flying Manolo’s hurt

    JD (134f7b)

  45. Economic theory tends to hold that greater choice leads to greater demand and consumption — thus, if you want to sell more, offer more choices to your consumers.

    Really? And what “economic theory” holds that this little jewel is true? (Or is “economic theory” being used instead of the concept, assertion?) Since Mr. Manzi’s experiment demonstrates that this obviously isn’t the case, one wonders how valid is the “economic theory” that is implied. I mean, no one ever bothered to test such an assertion before creating an “economic theory” around it? Sheesh…

    It’s garbage in, garbage out. Studies — even economic ones — can definitely prove certain things. However, the proof is all dependent upon how well the argument is constructed and to the degree from which the conclusion necessarily flows from the premises. Poorly conducted studies with broad, shaky conclusions are worse than worthless.

    Hey, logical arguments are hard to construct, man.

    J.P. (bd0246)

  46. This story (with the jams) is very famous.

    The more choices, the more time people take to make up their minds, and with something they don’t care for but are trying ouyt it seems people just think better of the whole thing.

    Sammy Finkelman (d22d64)

  47. Sheena Iyengar is the person who did the jam study. About maybe what half a year after she had published it she started hearing about this. Nobody knew she had done it.

    She wrote a book, in which this stiry of what it was is included,.

    The Art of Choosing by Sheena Iyengar. You can get it at Amazon.com (there are two other books with the same title one copyrighted in 1989 and one in 2005, but this ione is copyright 2011.)

    The Art of Choosing by Sheena Iyengar (Mar 9, 2011) – Bargain Price $6.00 Paperback (new)

    $6.19 used.

    In stock but may require an extra 1-2 days to process.

    $8.89 Kindle Edition
    Auto-delivered wirelessly

    More Buying Choices $13.99 Audio CD
    Hardcover $5.98 new $1.64 used.

    Sammy Finkelman (d22d64)

  48. Excerpt from page 183: … That’s how the jam study came about.

    Page 177 … CHAPTER SIX Lord of the Things I. STUCK I N A JAM H ave you hear…

    Page 178 … And such encounters with enthusiastic jam study dis

    Page 184 …250 mustards, 250 different cheeses, over 300 flavors of jam, and…

    Page 185 … between offering a large assortment of jams and a small one…

    Page 186 …for one week that knocked $ 1 off any single Wilkin & Sons jam ..

    Page 187 …people who had seen the small assortment decided to buy jam, …

    Page 190 … Since the publication of the jam study, I and other researc…

    Page 194 …that seem fairly benign, like wasting several minutes in the jam aisle…

    Page 195 …been gradually rising. He had recently read my paper on the jam study

    Sammy Finkelman (d22d64)

  49. Book review:

    http://www.nytimes.com/2010/04/18/books/review/Postrel-t.html?pagewanted=all&_r=0

    Wherever she goes, people tell Iyengar about her own experiment. The head of Fidelity Research explained it to her, as did a McKinsey & Company executive and a random woman sitting next to her on a plane. A colleague told her he had heard Rush Limbaugh denounce it on the radio. That rant was probably a reaction to Barry Schwartz, the author of “The Paradox of Choice” (2004), who often cites the jam study in antimarket polemics lamenting the abundance of consumer choice. In Schwartz’s ideal world, stores wouldn’t offer such ridiculous, brain-­taxing plenitude. Who needs two dozen types of jam?

    “The study hardly seems mine anymore, now that it has received so much attention and been described in so many different ways,” Iyengar, a professor at Columbia Business School, writes in “The Art of Choosing.” “From the various versions people have heard and passed on,” she adds, “a refrain has emerged: More is less. That is, more choice leads to less satisfaction or fulfillment or happiness.”

    Now Iyengar is having her own say about the jam experiment and the many other puzzles and paradoxes of choice. More choice is not always better, she suggests, but neither is less. The optimal amount of choice lies somewhere in between infinity and very little, and that optimum depends on context and culture. “In practice, people can cope with larger assortments than research on our basic cognitive limitations might suggest,” Iyengar writes. “After all, visiting the cereal aisle doesn’t usually give shoppers a nervous breakdown.”

    Sammy Finkelman (d22d64)

  50. #44, elissa, if the shoe fits…

    ropelight (59f33e)

  51. Chip S. says only a moron would think anybody fell for the jam experiment:

    I don’t think that more than a tiny number of economists looking for heterodox things to write put any stock in it.

    Yet this very thread is already jam-packed (thank you! that was indeed deliberate; thank you for noticing!) with links showing people believing the study is meaningful.

    And here it is in a book! Go here and put “jam” in the search box to see the study discussed.

    Remember: most studies actually show the opposite of what the jam study showed.

    Patterico (8b3905)

  52. Patterico, you’re exactly right. I have an M.S. and only one study does not mean much at all.

    Tillman (51d7aa)


Powered by WordPress.

Page loaded in: 0.0903 secs.