God Himself Appears in Human Form
You non-religious types must be feeling pretty stupid right about now. Because if you believe this AFP article, it appears that God Himself has reappeared in the world in human form — and it turns out that He is a statistics professor at Northwestern University.
US juries get verdict wrong in one of six cases: study
So much for US justice: juries get the verdict wrong in one out of six criminal cases and judges don’t do much better, a new study has found.
And when they make those mistakes, both judges and juries are far more likely to send an innocent person to jail than to let a guilty person go free, according to an upcoming study out of Northwestern University.
“Those are really shocking numbers,” said Jack Heinz, a law professor at Northwestern who reviewed the research of his colleague Bruce Spencer, a professor in the statistics department.
The shocking part to me is why we rely on the criminal justice system to settle factual disputes to begin with, when all the answers are already known by Northwestern statistics professor Bruce Spencer.
Also known as “God Himself.”
Think of the waste involved. In any given trial, at least 12 people (and usually two alternates) listen to evidence for days, and spend hours (often days) deliberating over a verdict. When they could have simply asked someone who knows the answer, with total, scientific certainty: Northwestern statistics professor Bruce Spencer.
Let’s review his “shocking” findings, as related by this completely unquestioning AFP article:
The study, which looked at 290 non-capital criminal cases in four major cities from 2000 to 2001, is the first to examine the accuracy of modern juries and judges in the United States.
It found that judges were mistaken in their verdicts in 12 percent of the cases while juries were wrong 17 percent of the time.
More troubling was that juries sent 25 percent of innocent people to jail while the innocent had a 37 percent chance of being wrongfully convicted by a judge.
And just how does some statistics professor sitting in his office know whether the people in these cases were truly guilty or innocent?
Spencer’s study does not examine why the mistakes were made or which cases ought to be overturned.
Instead, he determined the probability that a mistake was made by looking at how often judges disagreed with the jury’s verdict.
“If they disagree they can’t both be right,” he explained.
Spencer found an agreement rate of just 77 percent, which means a lot of mistakes were being made.
Oooookayyyy . . . I understand that if a judge disagrees with the jury’s verdict, the judge and jury can’t both be right. But just how, exactly, does this professor determine who was right, in order to come up with the “shocking” statistics mentioned above?
The unquestioning functionary who wrote this story seems to have no interest in exploring that question, or showing any of that vaunted journalistic skepticism we hear so much about. The conclusion fits the journalist’s preferred storyline. File it under “Too Good to Check.”
Don’t try to tell me that maybe the professor had some objective criteria to go on — like in each of the cases there were DNA results to show who did it. Even if that’s true, it merely shows that the sample is not reflective of criminal trials as a whole.
I haven’t seen the study itself, but I’m calling bullshit. There is, quite simply, no way that some statistics professor sitting in his office can know the true guilt or innocence of 290 criminal defendants. That’s why we have a system, pal — because there is no way for any one self-appointed individual to be the Sole Arbiter of who’s guilty and who’s not.
Unless, of course, Bruce Spencer really is God Himself, returned to Earth in human form. But if that’s really the case, then the End of the World is nigh — meaning we have a lot more to worry about than the accuracy of our criminal justice system.
Looks like another 402 motion for prosecutors.
Alta Bob (4fd45e) — 6/29/2007 @ 6:06 amOn the other hand maybe the professor is referring to the OJ verdict.
Alta Bob (4fd45e) — 6/29/2007 @ 6:08 amYou go Patterico. Love your site. It’s a must read daily. By the by this Spencer sounds like another tinfoil hatted leftist loon in academia doing harm to our children.
Sue (dd973d) — 6/29/2007 @ 6:28 amI saw that and thought the same thing…
I wasn’t aware that judges routinely announce their agreement or disagreement with a jury’s verdict, either. While I can believe that a judge forms an opinion during the course of a trial, it is the judge’s duty to accept a jury’s findings of fact (unless there is some procedural problem with how the verdict was determined), is it not? What’s up with this?
The idea that 25% of the people sent to jail by juries are innocent doesn’t pass the laugh test. I’d be shocked if it was 1/10th of that.
LagunaDave (69e495) — 6/29/2007 @ 6:42 am25% to 37% reversal on reasonable doubt is bullshit. Unless he preselected his cases in which case it’s fraud. I’m speculating that if he used an honest sample he’s counting multiple-count verdicts which were subsequently merged either as lesser includeds or by operation of “one act-one crime” statutes. Which is still a misleading conclusion.
nk (d0f918) — 6/29/2007 @ 6:49 amApparently the set of cases was a special one, in which the judge was given a questionnaire during the jury’s deliberations.
The Northwestern University press release, which the AFP report follows almost verbatim, but which has a few more details, is here.
The actual journal article is available here.
LagunaDave (69e495) — 6/29/2007 @ 6:59 amInstead, he determined the probability that a mistake was made by looking at how often judges disagreed with the jury’s verdict.A defendant is automatically “innocent” because the jury and judge don’t see eye-to-eye?
Wha…?
Darleen (187edc) — 6/29/2007 @ 7:47 amThanks Laguna Dave. I read the article and here’s my conclusion: If you carefully preselect a sample of cases, you can measure the times a judge disagrees with a jury in the cases within that sample.
A valid study can ve done very simply and at probably less cost. Track cases at random to reversal on reasonable doubt whether on direct appeal, collateral proceedings or executive clemency.
nk (d0f918) — 6/29/2007 @ 7:51 amThat’s not what the report claims. Judge and jury agreed in 77% of the cases studied. That could mean the jury was “right” 100% of the time and the judge was right 77% of the time, or vice versa, or it could mean each was right 87% of the time. Or any number of other combinations. But with 77% agreement, it is impossible that both are right more than about 87% of the time. The statistical analysis tries a variety of assumptions, and seems sound to me, as far as it goes. The question is the data.
The actual data used in the study is documented here and here.
The key is that this sample of cases (about 1/4 of which were in LA County, BTW) was originally selected to study hung juries. So the people who collected the data a priori chose jurisdictions with an unusually large rate of hung juries. It’s not clear whether other selection biases were involved. The judges and juries voluntarily filled out the questionnaires, but the response rate was approximately 90%. But the question I can’t answer from the articles is whether the cases where information was solicited were truly random.
LagunaDave (69e495) — 6/29/2007 @ 8:03 amI followed the link to the article. Thanks to Laguna Dave.
I didn’t have to read very far until I came on this sentence: “Yet, such direct assessments of accuracy are not possible on a wide scale because only atypically is the correct verdict known, and it is difficult to generalize from those cases to the more typical cases where the correct verdict is not knowable.”
Think about that: the typical case is one where the correct verdict is not knowable. But in the real world, you’ve got two sides who want a decision on disputed facts, whether it’s a civil or a criminal case. Somebody has to decide. You can’t just go home and say, “Well, the correct verdict is not knowable, so we’ll sit on our keisters and watch the flowers grow.”
And if the correct verdict is “not knowable” in the typical case, than how can a statistics professor at Northwestern know that the jury or the judge got it wrong? I mean even if you had the professor on call to decide your case, he doesn’t “know” the correct verdict.
There are a lot of things in life where you do the best you can and can’t be more than 99 and 44/100% sure that you got it right. You make a decision when you have to. Get over it and move on.
You can put this one down in the junk science/bull shit column. I gather the professor’s methodology involved having the judge answer a questionnaire about what he thought of a jury verdict. I’d like to read what Judge Ito thought of the O.J. verdict.
Mike Myers (2e43f5) — 6/29/2007 @ 8:03 amThe actual article includes a lot of caveats about the data being atypical that are glossed over in the press release and the AFP report. But since Spencer presumably signed off on the press release (which suggests more generality than is warranted), I suspect he was probably trying to have it both ways.
Actually, I think the only way to do a really believable study would be to have two (or more) juries hear the evidence in the same courtroom, deliberate independently, and reach a verdict. Then you could assess reliability of the verdict by comparing one jury with another. If it were done as a controlled experiment, the jury whose decision would represent the “real” verdict would be determined randomly in advance, and double-blinded from the participants in the trial. You would also have to prevent the jurors from different juries from communicating with each other, at all. But of course then we would be talking about a different system, so it’s not very practical.
Another important thing to keep in mind is that the judges and jury are working with different knowledge. In particular, the judge likely knows many things that the jury is not privy to because they were not admitted into evidence. And since the rules of evidence (IMO) are primarily designed to protect the defendant, it isn’t surprising that judges, who may not be able to perfectly filter out formally inadmissable information, are more likely to convict than juries.
LagunaDave (69e495) — 6/29/2007 @ 8:27 amObviously, we need to disband the jury system and appoint a few brilliant, dedicated leftist professors to manage this whole guilt/innocence thing. Perhaps a special municipal department could be instituted?
Patricia (824fa1) — 6/29/2007 @ 8:28 amI think Bruce Spencer has it about right. I make similar judgments all the time. I watch or read about decisions and come to my own conclusion. I haven’t written down my findings, however. If I had, I could have been (in)famous too. Of course, I’m not a college professor, so the media wouldn’t care–I’m not an ‘expert.’
I think my procedure would carry about as much weight as Bruce’s, don’t you?
Hey, let’s all vote on whether we believe each verdict, having heard none of the evidence.
ManlyDad (22e85d) — 6/29/2007 @ 8:38 amCouple of things. As much as I’m no fan of jury nullification, juries do what they think is right. Sometimes that may or not square with someone’s notion of correct result. Defining “correct result” is therefore the first problem.
Second, juries provide the community’s reaction to the issues placed before them. Rarely do they leave without being convinced they have arrived at a correct result as far as their community is concerned.
Third, where I come from it is unethical for judges to comment on cases except in the rarest of circumstances. I assume the researcher was able to work around those concerns as he compiled his data.
Last, someone’s opinion who did not sit through the entire trial is less valuable (and in many cases, worthless) than anyone who did. The weight to be given to opinions expressed by sideliners is something that should not be overlooked in these kinds of discussions.
Ms. Judged (becd1d) — 6/29/2007 @ 10:12 amI don’t get it. What’s the problem with this study? As far as I can tell, the author is simply stating that when two independent observers (judges and juries) are asked to judge the merits of a criminal case they often come to opposing conclusions. That’s quite striking, and it is suggestive that there may be more incorrect decisions (both wrongful conviction and acquittal) than I expected. Both the paper and the press release include the caveat that the case sample may not be representative and “that the numerical findings should not be generalized to broader sets of cases, for which additional study would be needed…” At the very least, this paper suggests that continued study would be worthwhile. It seems to me that most people would agree that the criminal justice system makes mistakes on occasion, and it suits all of us to try and eliminate as many of those mistakes as possible. So, the more different ways we have of studying the problem, the better off we are.
adams (40d1a3) — 6/29/2007 @ 10:29 amPat, I’m shocked that you’ve not heard of Bruce Almighty!
Harry Arthur (b318a5) — 6/29/2007 @ 11:10 amSome points, for what they are worth:
1- Bullshit/Junk Science – most likely. Newsies are particularly vulnerable to this crap – might have something to do with the VERY low SAT/ACT scores they usually put up, and the dearth of hard-science curriculum in J-School;
2- Was this paid for with a Government grant? Boy, I hope not, but wouldn’t be surprised;
3- Is this study now going to be the basis for another grant request because it just cries out for further study? (my impression is that most of this junk science crap we see in the papers – or brought to us breathlessly on TV – are a continuing quest for funding by “academes” who are completely incapable of surviving in the real world);
4- Could we ask this guy what the “correct” verdict is for the cop-shooting trial that just concluded in SanBerdoo County (cop acquitted – someone will have to ‘splain that one to me)?
Another Drew (8018ee) — 6/29/2007 @ 11:19 amHeh, funny, informative take, P. This is what I keep you around for. 🙂
David N. Scott (71e316) — 6/29/2007 @ 11:38 amThat they agreed in only 77% is astonishing to me (granted, this is a newspaper article that has numbers in it, so it must be treated with very great suspicion!) Noticing that the prof seems to use actual statistical methods, I want the weekend to read the paper. I wonder if he’s going to go into the “did the deed” vs “should be convicted” difference?
htom (412a17) — 6/29/2007 @ 11:52 amAnother Drew’s (#16) comments make the point I talked about earlier. The people’s burden is to convince the jury of the truth of the charges “beyond a reasonable doubt”. If they fail to meet that burden, they are obligated under California law to acquit. The jury’s decision as to whether the people have or have not met their burden is necessarily subjective. The choice is their’s and their’s alone as is the criteria and evidence they choose to rely upon in reaching a decision. If they come to the conclusion that the people’s burden was not met, the required verdict of not guilty is the correct result.
Back to Another Drew. S/he obviously thinks that the correct result would have been to convict the ex-deputy. After all he shot the guy. Right? But as I just said, that’s not the right question in the legal sense. It’s the jury’s call. Not guilty? Done deal. Finitio.
The people cannot appeal a verdict of not guilty. The decision therefore is not subject in the legal sense to a “correct” or “incorrect” analysis by any court.
A defendant can appeal many aspects of a guilty verdict. Many are overturned on appeal. But the overwhelming majority of reversals have more to do with errors in certain procedural aspects of the trial, e.g., improper introduction or suppression of evidence, improper jury instructions, ineffective assistance of counsel, etc. Rarely do appellate courts announce that the result is incorrect and work backwards looking for a reason(s) to justify the conclusion. Even then, in most cases, the appellate courts are required to interpret cases in the light most favorable to the jury verdict.
Was O.J. guilty? Of course. Was the jury incorrect in finding him not guilty? No. It hurts and strains credulity, but that’s how it is and that’s what makes this study and the results described very questionable.
Ms. Judged (becd1d) — 6/29/2007 @ 11:52 amIsn’t it likely that the judge heard more evidence than the jury did, such as proffered evidence that was excluded or limited as irrelevant or due to evidentiary rules? Even if a judge tries to disregard irrelevant and inadmissible evidence, to keep it from affecting his/her conclusion on the merits, it would be hard to disregard everything heard and seen. Perhaps this plays a role in why some judges reach different conclusions than the juries.
DRJ (31d948) — 6/29/2007 @ 12:18 pmDRJ: I believe that in the study the author says that judges were slightly more likely that juries to report that a defendant was guilty. This is probably due to the factor that you mention, in that they often see more evidence than the juries.
adams (40d1a3) — 6/29/2007 @ 12:29 pmThanks, Adams. Fortunately you are a more careful reader than I am!
DRJ (31d948) — 6/29/2007 @ 12:45 pmon page 2 of his paper, right in the middle, he says:
“In this paper we show that an estimate of jury accuracy can be derived from the observed agreement rate even when the correct verdict is unknown.”
Now, you’re probably not going to understand how. So stick with superstition, and put it on God.
whitd (a649f3) — 6/29/2007 @ 1:06 pmWhitd,
Rather than the words you bolded, I think the critical word is: “we show that an estimate of jury accuracy can be derived …”
DRJ (31d948) — 6/29/2007 @ 1:24 pmI think the process by which jurors are chosen is called voir dire and that the process seeks to exclude those who have opinions about guilt/innocence before they’ve heard the evidence.
Isn’t that correct? You lawyers can feel free to correct any errors with aplomb about your ‘speciality’, don’t you? And well you should.
However, I don’t believe the report is in evidence yet, is it? Did any of you find it in it’s entirety? Please direct me to the link if I am mistaken.
Over 200 cases were studied by a STATISTICIAN.
That’s his specialty. Are you statisticians?
” Recent high-profile exonerations of scores of death row inmates have undermined faith in the infallibility of the justice system, Heinz said.”
That may be what was concluded through DNA taken from inmates which, as you know, has been resulting in quite a few exonerations. Now, you could say that God, as the author of DNA, is involved, but it doesn’t take God to read stats
when you are trained to do so. But, again, I don’t know what his evidence is yet, nor do you.
So lighten up, lawyers.
Semanticleo (10a7bd) — 6/29/2007 @ 2:02 pmI have my doubts about the study too, but it appears that prosecutors do the same thing but in the opposite direction. They often seem to think that the system is almost perfect and can rarely conceive that the system makes mistakes–many more mistakes than most prosecutors are willing to admit.
Daniel Quackenbush (262c6b) — 6/29/2007 @ 2:46 pmDaniel,
I’m not a prosecutor but your statement seems painted with a pretty broad brush. In fact, I might even say you’re spray-painting.
Prosecutors in my town are very concerned with innocence and justice. In cases where the evidence is not clear, I’ve seen the DA’s office re-investigate a case and come to a different conclusion than the police reached. In fact, I’ve even seen the DA personally re-interview witnesses (of which I was one) so he could decide for himself if it was appropriate to pursue a case. In that case, the DA decided not to pursue charges even though I’m confident he could have won the case. Instead he dropped all charges because he was not convinced of the defendant’s guilt.
DRJ (31d948) — 6/29/2007 @ 4:55 pmDRJ,
You are right. There are some excellent and ethical prosecutors out there. However, it is my experience that prosecutors *often* are not out to see that “justice is done.” E.g., when a police officer uses excessive force or makes a false arrest, the prosecutor will often prosecute anyway so that the case Heck v. Humphrey (US S.Ct.) comes into play to protect the officer from a future false arrest lawsuit. In my daughter’s case, when the public defender said he wanted to make a motion to suppress, the prosecutor immediately threatened to increase the charges (which although factual, were not a crime). This DA became a judge about a year later by appointment.
Daniel Quackenbush (262c6b) — 6/29/2007 @ 5:12 pm#27 Daniel,
California DOJ statistics show that in 2005 328,663 felony arrests were referred to the LA County DA by police agencies. The DA released 21,490 of them.
Stu707 (5b299c) — 6/29/2007 @ 5:17 pm“Rather than the words you bolded, I think the critical word is: “we show that an estimate of jury accuracy can be derived …”
I know. I’m literate and numerate. I got that from this:
“Instead, he determined the probability that a mistake was made by looking at how often judges disagreed with the jury’s verdict.”
whitd (57aa5a) — 6/29/2007 @ 5:49 pmYes, I know that DAs don’t prosecute every case. It would be impossible with the number of false arrests police make to prosecute every case. Further, America, with it’s world population share of 5%, incarcerates 75% of the world’s prisoners, making it impossible to prosecute every case. There’s not much room left for them.
One time I tried to make a complaint against “one of your [DA’s] police officers” with the local DA, and the DA wouldn’t even interview me. He had the nerve to complain that they are not “his” police officers. In some places, for a DA to go against the police is political suicide.
Some of the “biggest” criminals we have in the U.S. are police officers, yet DAs rarely prosecute them (I don’t really blame them because it is difficult to get a conviction). This article is accurate: http://teamliberty.net/id224.html , yet most DAs are not interested in doing anything about it.
Daniel Quackenbush (262c6b) — 6/29/2007 @ 6:06 pmI think it’s the cell phones.
Kevin Murphy (0b2493) — 6/29/2007 @ 6:23 pmThe study could not possibly show what it purports to show. Bruce Spencer, the study’s author, claims “Some of the errors are incorrect acquittals, where the defendant goes free, and some are incorrect convictions…. As a society can we be satisfied if 10 percent of convictions are incorrect? Can we be satisfied knowing that innocent people go to jail for many years for wrongful convictions?”
None of us believes that 100% of convictions are accurate, but this study does not preclude the possibility that all of the errors are acquittals of the guilty. Indeed, it is probable that many fewer than 10% of convicted defendants are innocent.
Further, as Spencer concedes, this study makes a lot of “assumptions.” One of his unstated assumptions is that letting a guilty person off is a mistake, even if there is insufficient evidence of his guilt, or if some of this evidence is inadmissible.
People are not perfectly rational, of course, but even perfectly rational judges and juries with, say 1% error rates, are compatible with a 23% difference in judge and jury verdicts.
First, we have, as mentioned before, the fact that judges know things that juries do not, and yet as a society we have chosen that these things not be known to juries.
But let us exclude even this possibility from the equation. The study still does not prove error rates in a meaningful sense which are close to half of 23%.
Each rational juror and judge must come to some conclusion about what they think are the chances the defendant actually committed the crime, and also what they think are the chances which correspond to guilt “beyond a reasonable doubt.” If the former is higher than the latter, his verdict is guilty.
Of course, the law has no mathematical standard for reasonable doubt. A lot of variation between individual judges and jurors could come from a varying level of this standard. But let us exclude even this possibility from the equation. Even if everyone had the same such mathematical standard of guilt, the study’s 23% differential outcome would still be compatible with a much lower rate of erroneous conviction.
Let us suppose that a given juror or judge believes that the notion of “reasonable doubt” requires 98% certainty of guilt, or a 2% chance of innocence. We know that 23% of the time the judge believes that a given defendant’s odds of actual innocence are above 2%, while the jury’s opinion is the opposite (or vice-versa). But suppose in the typical such case the judge decided there was a 3% chance of actual innocence, while the jury thought 1%. This sounds pretty close, but we are talking about cases where opinions differ, not the obviously innocent or guilty, who are often not prosecuted, or plea bargain, or go to trial where the judge and jury agree. It is also compatible with a 1% standard deviation in a judge or jury’s estimate of the chance of innocence.
Obviously, if there are 1000 cases, and 23% of them, or 230 people fit an example comparable to the above, then that’s 230 people who could either be convicted or acquitted, depending on whether they had gone with a jury or a judge. But with people looking to acquit anyone with a 2% chance of innocence, even if the judge is right, and a person who should have been found not guilty was instead convicted by a jury, then only 3% of these 230 people, or 7 people, are wrongly convicted, and that is only 0.7% of the 1000 cases being wrongly convicted.
A simpler way of looking at it: If people’s notion of “beyond a reasonable doubt” is to require 98% certainty of guilt, then we would expect fewer than 2% of the convicted to be innocent (1%, if the distribution of guilt possibilities is linear). And if people’s notion of “beyond a reasonable doubt” is to require 90% certainty of guilt, as Blackstone would have it, we would expect about 5% of the convicted to be innocent. If the latter number is “too high” to you, we could change the definition of reasonable doubt either with stricter-sounding but subjective terms, or with something more specific. But we would of course then have the higher costs of more guilty set free. Further, even rational people looking to impose a 99.99% guilt standard could disagree 23% of the time. (In this case, almost all of the disagreements of course being over people who in fact committed the crime.)
As far as the error of letting off a defendant who is actually guilty, but where the evidence was lacking, that is not a jury error, but rather the result of the Blackstonian imperative, as well as the rules of evidence. It is also not much different and no worse than the far more numerous cases where criminals are never caught, never arrested, or never prosecuted.
DWPittelli (2e1b8e) — 6/29/2007 @ 7:39 pmDaniel,
Your family has had a bad experience and it has obviously affected your views. I’m sure the Duke lacrosse players and their families could identify with your experience. However, I think it’s unreasonable and unfair to use words like “often,” “most,” and “some of the biggest criminals we have” to describe prosecutors and DAs and, if you want to be taken seriously, then you should link to credible statistics that prove your point.
DRJ (31d948) — 6/29/2007 @ 7:39 pmDWPittelli,
Good comment.
DRJ (31d948) — 6/29/2007 @ 7:42 pmYou are right. My perception is *partly* based on my experiences both as a civilian and a cop for about 6 years (plus about 2 years as a correctional officer). I know for a fact that when the police are “out to get you,” you will come upon such blatant perjury that it will scare the bejesus out of you. The problem of “police crime” and corruption is greatly underreported and rarely prosecuted. Thus, when people claim there is not much police crime, I know they don’t know what they are talking about.
Daniel Quackenbush (262c6b) — 6/29/2007 @ 7:52 pmAn example of the difficulty of prosecuting police and correctional officers is the California Corcoran prosecutions of “guards” a few years ago. Every night I would hear on the radio how the Corcoran correctional officers “walk the toughest beat in the state,” a blatant attempt at extrajudicial influencing of public opinion and the jury. You don’t want to get me started because I can tell you of police planting of drugs, and all kinds of illegal police misconduct. When I was a cop, I heard police telling rookies how it is OK to lie (or plant evidence, etc.) because certain criminals just hadn’t been get caught before so it doesn’t hurt to lie to get a bad guy off the street. Police, in general, don’t like niceties of the Constitution, such as the Fifth Amendment right to remain silent, or the right to be convicted based on only truthful testimony.
Daniel Quackenbush (262c6b) — 6/29/2007 @ 8:08 pmWow using this thought process the American electorate must be wrong an amazing amount of the time every time it turns out an incumbent. I mean what does the incumbent think as compared to the electorate? They can’t both be right can they? Therefore according to the professor democracy can’t be working correctly.
Save us from academics.
Thomas Jackson (bf83e0) — 6/29/2007 @ 9:13 pm“I mean what does the incumbent think as compared to the electorate? They can’t both be right can they? Therefore according to the professor democracy can’t be working correctly.”
You’re not really serious are you? This is a situation where there is one answer — guilt or not. Not so with elections.
whitd (285749) — 6/29/2007 @ 9:28 pmDWPittelli: I think your comment is pretty much right on, and that’s the scary thing. This data suggests that there is a very large number of defendants for whom the evidence puts them on the cusp of reasonable doubt. So much so that two different observers (judge and jury) come to differing conclusions. It seems to me that it’s important to learn more about how and why these different conclusions are determined. A first step in studying this is determining the magnitude of the issue, and I think that’s where studies like this come in.
adams (233b7c) — 6/29/2007 @ 9:50 pmDaniel,
I appreciate that you have provided your impressions about law enforcement officers, and I assume from your outrage that you weren’t that kind of officer. Good for you. I’m sure there are many professionals in law enforcement who share your concerns and values.
DRJ (31d948) — 6/29/2007 @ 9:52 pmMost good cops won’t admit they turn a blind eye to the corrupt cops because they get comments like yours, among worse other things. (That’s an example of why good cops can’t turn in bad cops without danger to either their reputations or lives.) The only moral justification a good person can have to remain a cop nowadays is that if they weren’t “on the force” then the really bad cops would “take over.” You have to actually be a cop to know how it is impossible for a good cop to turn in a bad cop in most departments and not be harassed by other cops or not get backup when the good cop needs it the most. To remain a cop for long, that cop can’t turn in other cops. (Ever wonder why cops are never arrested when they illegally assault a person–it’s almost always based on a secret videotape–such as the recent Chicago police officers caught on tape beating the heck out of a woman bartender?) Did you see the recent video of the police in N.Y. threatening to accuse a protester of having a bomb so the protester would have to spend a few days in jail until it was “all sorted out?”) Good cops ordinarily won’t get the support of police “management,” and you can be assured that ordinarily the DA won’t believe a word the good cop says, won’t care, or will realize that prosecuting a bad cop is politically hazardous.
You will rarely find a cop that is still working as a cop who can expose police corruption.
Daniel Quackenbush (262c6b) — 6/29/2007 @ 10:22 pmwhitd,
“This is a situation where there is one answer — guilt or not”
Actually, no. The question is “Is there enough proof of guilt within the standard for conviction?”, and that’s just as subjective as “Did Senator Joe Bob do a good job?”.
Dave (6001a6) — 6/29/2007 @ 10:31 pmAnother example of how DAs “overlook” corruption is illustrated here:
“SB 111 would have made public all transcripts from grand jury trials in officer-involved deaths; since grand juries are run by district attorneys, and since there hasn’t been a successful case against an officer in pretty much forever, there is widespread belief that the testimony and evidence given to grand juries is weighted in the officer’s favor. This is only the latest effort—and latest failure—to get those transcripts made public, in order for the community to gauge the fairness and accuracy of the trials.”
http://blogtown.portlandmercury.com/2007/06/still_no_grand_jury_disclosure.php
As I said before I don’t really blame the DAs for doing what is alleged above because, among other things, police have their ways of ensuring the defeat of any prosecutor who gets “too tough” on police corruption. You may have noticed the coveted “we are endorsed by the Deputy Sheriff’s Association” type ads when DAs are running for election. Here, even judges running for election will often list in their campaign literature how they have the support of certain police/sheriff’s associations, etc.
Daniel Quackenbush (262c6b) — 6/29/2007 @ 10:50 pmDaniel,
I have two relatives that are law enforcement officers. They share your cynical outlook about some aspects of their profession but, in general, they are proud of the way they and their fellow officers do their jobs. In addition, both acted to expose wrongful conduct and corruption when they saw it.
If you and some of your colleagues refuse to do what you know is right, it’s because you made that moral decision and not because you had no choice. That’s true for doctors, lawyers, teachers, and everyone else. People in all professions sometimes do bad things but if you want to believe your profession is the only one that’s morally bankrupt, I guess that’s your prerogative.
DRJ (31d948) — 6/30/2007 @ 4:13 amNoting that 80% or 90% of the discrepancies involved the judge finding guilty and the jury finding not guilty, this discrepancy pattern can be explained by 3 factors:
1) The factor suggested earlier (that judges often know things they have not allowed juries to hear, and judges may be unable or unwilling to completely discount such inadmissable evidence in otherwise close cases).
2) The possibility that juries may have less respect for the testimony of police.
3) The possibility of jury nullification, when the jury either disagrees with the existence of a statute, believes a guilty verdict will result in an unjustly harsh punishment, or thinks the state acted improperly in some way in the course of entrapping, detecting, investigating or prosecuting the crime.
In all three cases, the discrepancy, which leans toward the acquittal of defendants who actually broke the law, may be seen as a feature, not a bug. We have juries precisely so that the criminal justice system can’t throw people in prison without attaining the unanimous consent of a group of ordinary citizens.
DWPittelli (2e1b8e) — 6/30/2007 @ 4:30 amAlthough I have seen equally egregious cases, it is is still pretty shocking to see that an academic social scientist could get 33 pages published out of such weak gruel, with all sorts of “sophisticated” statistics claimed to show what they cannot, and claimed to confirm what was obvious anyway. So much intelligence and so little common sense.
The author needs reams of stats to conclude that “The estimates tentatively suggest that the type I error rate [conviction of the innocent] is lower for juries than judges,” when the mere fact that most of the discrepancies involve the jury finding not guilty and the judge finding guilty shows this right off the bat.
He also seems to think that if he puts the word “estimates” in italics, then that excuses complete B.S. as he comes to the most dubious of conclusions about how often each type of error is made, and what portion of defendants are actually guilty.
DWPittelli (2e1b8e) — 6/30/2007 @ 4:47 am“Actually, no. The question is “Is there enough proof of guilt within the standard for conviction?”, and that’s just as subjective as “Did Senator Joe Bob do a good job?”.”
But “good job” means different things to different people. THey have different preferences about what they want their goverment to do. Standards of guilt aren’t supposed to mean different things to different people. They’re supposed to be the same.
“The author needs reams of stats to conclude that “The estimates tentatively suggest that the type I error rate [conviction of the innocent] is lower for juries than judges,” when the mere fact that most of the discrepancies involve the jury finding not guilty and the judge finding guilty shows this right off the bat.”
Not really ‘off the bat’ because ‘off the bat’ the jury could be making type II errors and the judge is right. You need more data or assumptions or constraints to determine which is which.
“He also seems to think that if he puts the word “estimates” in italics, then that excuses complete B.S. as he comes to the most dubious of conclusions about how often each type of error is made, and what portion of defendants are actually guilty.”
You’re joking right? Estimates means something in statistics. This is complicated stuff. But you know enough to say it is BS? Because you think its “sophisticated”?
At least you don’t think its god.
whitd (e1d099) — 6/30/2007 @ 6:15 amI had a long screed in mind, but DWPittelli has hit the ball square and hard. Reasonable jurors and jurists can differ; my version of reasonable doubt may not follow my fellow jurors, and my doubts as to guilt may not match, but we’re dealing with a class of people who are by and large guilty.
This is the primary source of disagreement. People who are found not guilty are seldom not actually guilty. (It happens, yes. But the guilty go free sometimes.)
On a side note, I don’t see what Mr. Quackenbush sees. I’ve prosecuted a cop (based on other cops’ statements), and I had a run-in with another cop over…. well, it was bad. Cops generally still treat me well.
–JRM
JRM (355c21) — 6/30/2007 @ 9:18 amwhitd,
No I am not joking. I am seriously calling this study a bunch of BS. Statistical “estimates” are BS when they are based on BS assumptions.
There is nothing in the data incompatible with there never being a conviction of an innocent person. There is nothing in the data incompatible with there never being an acquittal of a guilty person. There is even nothing in the data incompatible with the jury never making either kind of error. (Or of the judge never making making either kind of error; but of course, either the judge or the jury, or both, must make some errors, as there isn’t total agreement.) Indeed, there is nothing in the data incompatible with 100% of the defendants being actually guilty as known to an omniscient viewer.
All its claims to the contrary notwithstanding — some of them quite outrageous if true — there is nothing shown by this study more definitive, or even all that different quantitatively, than a seat of the pants calculation amounting to splitting-the-difference between judge and jury.
I will concede, as you suggest, that if the judge is making no Type I errors, then the juries’ rate of Type I errors cannot be lower than 0.
DWPittelli (2e1b8e) — 6/30/2007 @ 6:42 pmIn addition to the obvious problems identified in the main post, and the sample problems identified by LagunaDave, the notion that fewer than 300 non-capital criminal cases in four areas is somehow a representative sample of the American criminal justice system is laughable.
For example, in .2002, State and Federal courts convicted a combined total of about 1,114,000 adults of felonies. So we’re already excluding misdemeanor offenses.
Now let’s guesstimate that in the period actually studies, it was considerably less — a round 1 million.
If we use the standard polling confidence interval of 3 percent, and the standard confidence level of 95 percent, we can calculate that we would need a sample size of over 1,000 cases.
271 is a wee bit short of 1,066.
Karl (f95d06) — 6/30/2007 @ 7:08 pmIn addition to the problems identified in the main post and by lagunaDave, the notion that 271 cases would be an adequate sample, given the number of criminal cases prosecuted annually, is a joke. Using standard confidence intevals and levels, the sample would need to be over 1,000.
Karl (f95d06) — 6/30/2007 @ 7:12 pm“All its claims to the contrary notwithstanding — some of them quite outrageous if true — there is nothing shown by this study more definitive, or even all that different quantitatively, than a seat of the pants calculation amounting to splitting-the-difference between judge and jury.”
Seat of the pants? did you read the equations? did you understand them? How long did it take you to read section III of the paper?
Did you notice that he makes assumptions which lead him to his results? Besides assuming certain statistical models, he also assumes that judges are going to be at least as accurate as juries.
Still its a bunch of BS? did you understand those assumptions were even made?
whitd (394243) — 6/30/2007 @ 7:34 pm[…] Read the rest… […]
Jury Experiences (f515ed) — 7/1/2007 @ 2:22 amwhitd,
You are making my argument for me. Of course I understand that he “assumes that judges are going to be at least as accurate as juries.” That is the base problem: When the assumptions are pulled out of your ass (or that of a male bovine), bullshit is the result.
If he had instead assumed that the judges are exactly as accurate as the jury that would be exactly the splitting-the-difference I referred to, which would be no less justified an assumption, and would lead, with 1/100 the mathematical work, to equally robust (i.e., not very) and not dissimilar results. (It would not, of course, lead to a published 33-page paper.)
A statement can be literally true when it says, “if we assume X, then Y is probable,” but still bullshit if assumption X is bullshit. Further, the Y claim (e.g., 10% false conviction rate), when separated from the list of assumptions, remains false.
DWPittelli (2e1b8e) — 7/1/2007 @ 3:51 am“When the assumptions are pulled out of your ass (or that of a male bovine), bullshit is the result.”
I don’t understand your dislike for this. The result is as as strong as the assumption. Up above people were discussing why judges might be more accurate.
And again, are the calculations really “seat of your pants”? did you read them? how long did it take you?
“A statement can be literally true when it says, “if we assume X, then Y is probable,” but still bullshit if assumption X is bullshit”
So if the assumption is ‘reasonable,’ then this result is ‘reasonable’?
whitd (57e439) — 7/1/2007 @ 7:33 amwhitd,
I tried reading the paper but got lost in the equations. Maybe you could answer one question for me: does the professor purport to answer the question of the verdicts’ “correctness” in terms of the omniscient perspective? In other words, does he purport to use statistics to tell whether these people really are guilty, in the absolute sense? Because that’s what I assert he can’t do with statistics — and that’s what he appears, from the article, to be claiming to do.
Patterico (2a65a5) — 7/1/2007 @ 7:41 am“Maybe you could answer one question for me: does the professor purport to answer the question of the verdicts’ “correctness” in terms of the omniscient perspective?”
No. He just estimates probabilities of correctness by using the statistics for how often disparites occur and some assumptions. One of those assumptions is that judges are at least as accurate as juries. Other assumptions are the statistical and probability models he uses.
He does not know, nor does he claim to know, for any particular case what the correct result is. That is what I bolded in my first comment — that all of this is done without knowing the “true” result of any case.
His result is as strong as those assumptions.
shine (254a04) — 7/1/2007 @ 8:21 amHe does not know, nor does he claim to know, for any particular case what the correct result is.
But you didn’t answer my question. I know he doesn’t claim to know it for any particular case. But does he claim to know this *collectively*? Because the article — and press release — damn sure read that way.
Patterico (2a65a5) — 7/1/2007 @ 8:51 amAnd that’s what’s so absurd. Because even collectively, he just can’t know that — or even approximate it.
Patterico (2a65a5) — 7/1/2007 @ 8:52 amThe assumption that judges are more “correct” than juries …. After twenty-five years (and in some of those years a yearly caseload bigger than the sample in this study) I can name at most ten judges before whom I would unhesitatingly waive jury trial. In fairness, I can only name one judge whom I would always SOJ. As good as a judge may be, he is only one human being. I believe that the jury is greater than the sum of the twelve jurors. It will catch things the lawyers themselves will miss. It will never forgive fraud but it will forgive and even sympathize with a less than perfect presentation. A half-way normal jury will take its job seriously and try to do justice as opposed to going on a power trip or just wanting to get the trial over so the jurors can go back to their lives.
(The O.J. Simpson jury seems to come up a lot in these discussion. The fault is not with the jury but with Judge Ito. He totally failed to protect them. He essentially jailed them for the duration of the trial. He allowed hundreds of hours of irrelevant, collateral, prejudicial and downright tedious testimony. I think that if the jury had been given the option to imprison Ito they would have.)
nk (d0f918) — 7/1/2007 @ 9:02 amwhitd: “are the calculations really “seat of your pants”?”
I do not claim the calculations are seat of your pants. I claim their results are no more accurate than one could get from seat-of-pants calculations.
“So if the assumption is ‘reasonable,’ then this result is ‘reasonable’?
Actually, the judge being more accurate than the jury is only one of the article’s assumptions. If all of its assumptions, stated and unstated, are reasonable, then the result is probably reasonable. (The .pdf refers often to “different assumptions” without always stating them.)
DWPittelli (2e1b8e) — 7/1/2007 @ 9:48 amPatterico,
Although the study author claims that his estimated “accuracy rates apply to both the “procedural” and the “omniscient” interpretations of correct verdict,” I do not believe that his accuracy rates are any more reasonable for either interpretation than they would be just assuming some weighted midpoint between a judge’s and jury’s verdict. That is, we know when judge and jury disagree. If we assume that the judge is right, say, 60% of the time, we then know a minimum error rate for the jury (minimum because both judge and jury can be wrong in a given case, and thus in agreement). This error rate is necessarily independent of our definition of accurate (e.g., omniscient vs. procedural).
DWPittelli (2e1b8e) — 7/1/2007 @ 10:15 am“And that’s what’s so absurd. Because even collectively, he just can’t know that — or even approximate it.”
Yes he can. He can come up with bounds and fit it to probability models. Because he doesn’t just have the agreement /disagreement rate. He also has his assumptions. He explains it in his paper between pages 6 and 12. Later in the paper he comes up with a different model. Right now it’s peer reviewed math you don’t understand. Nothing absurd about this.
Note that he doesn’t even know collectively how many cases are correct. All he knows is a probability.
“I claim their results are no more accurate than one could get from seat-of-pants calculations”
How accurate is that? Whats the percent and sources of error?
whitd (41fec7) — 7/1/2007 @ 10:37 amHe also has his assumptions. He explains it in his paper between pages 6 and 12. Later in the paper he comes up with a different model. Right now it’s peer reviewed math you don’t understand.
I don’t understand the math but I can understand assumptions expressed in English. I read the one about judges’ verdicts being as accurate as juries’ — but that doesn’t appear to relate to the omniscient perspective, since it’s all based on the evidence in the courtroom.
Please explain to me in English how he gets to aggregate conclusions based on the omniscient perspective. And don’t give me this “you couldn’t understand because you don’t know math” excuse. If you do that, I’ll consider it a dodge and discount it utterly. If you really understand it, you can explain it, or at least try, using English.
Patterico (2a65a5) — 7/1/2007 @ 10:47 amNote that he doesn’t even know collectively how many cases are correct. All he knows is a probability.
But on what basis does he claim to compute this probability based on evidence heard in the courtroom? Is one of his assumptions that the “legally correct” verdict is roughly approximate to the correct verdict using the omniscient perspective?
Patterico (2a65a5) — 7/1/2007 @ 10:49 amwhitd,
The accuracy is indeterminable, because the key source of inaccuracy is not the sample size, or otherwise calculable. The key source of inaccuracy is the unwarrantedness of the assumptions.
In addition to assuming that judges are somewhat more apt to be right than juries (which right there brings him most of the way to his shocking conclusion about errors), the author also assumes that the data is meaningful, that the strength-of-evidence question is answered in a consistent way by all parties, and that disagreement over whether the burden of proof has been met means a comparable level of error in the verdict by the omniscient definition.
His data is the 2 x 2 matrix of whether the judge and the jury found the defendant guilty or not; and for the NCSC cases, whether the judge and the jury believed the evidence to be weak, medium or strong. If you look at that limited and subjective data and assume that it contains his claimed objective truths which can be wrung out with statistical study, you will be able to come up with such an objective-looking conclusion. That doesn’t make it accurate. GIGO.
To me, it is obvious that such data cannot contain the objective truths claimed for it, and that no amount of statistical modeling can make it otherwise. Perhaps that is my own arrogance speaking. But notably, and as I have mentioned earlier, there is nothing in the data inconsistent with the model of rational juries consciously working with a 99% certainty (or any other) standard for a guilty verdict, which by definition would mean that somewhat more (perhaps 99.5%) of the convicted are in fact guilty under the omniscient standard.
DWPittelli (2e1b8e) — 7/1/2007 @ 1:07 pmRight on, DWP.
Patterico (2a65a5) — 7/1/2007 @ 2:04 pm“Please explain to me in English how he gets to aggregate conclusions based on the omniscient perspective.”
he makes enough assumptions (including model choice) and has enough data to calculate an estimate of probability. I don’t understand his equations perfectly, but they involve things like assuming what sorts of effect independence of results would have, as well as what to use to model the probabilities.
“And don’t give me this “you couldn’t understand because you don’t know math” excuse. If you do that, I’ll consider it a dodge and discount it utterly.”
Thats ridiculous. I don’t fully understand his math. I’m just trying to convince you that this isn’t absurd or godlike.
“But on what basis does he claim to compute this probability based on evidence heard in the courtroom?”
As far as I can tell he doesn’t base it on evidence heard in court at all. Though he does have data on the relative strengths the evidence.
“If you look at that limited and subjective data and assume that it contains his claimed objective truths which can be wrung out with statistical study, you will be able to come up with such an objective-looking conclusion”
It contains an estimate of objective truth. You don’t know that the objective truth of a coin flip is that the chance of heads and tails is .5. But you can estimate it for a whole bunch of coins and several people flipping them. Subjectively.
whitd (34bc65) — 7/1/2007 @ 3:39 pmSorry, DWPittelli has the better of the argument.
Patterico (2a65a5) — 7/1/2007 @ 7:24 pmIt is sorry. He doesn’t seem to know the difference between an estimate and an objective truth. And you? you’re stuck on that too. Because to begin with, you imagined that the only way to possibly publish this was to also know the objective truth. You never imagined how statisticians work to estimate probabilities. So along comes someone and reinforces that.
Whats the argument? That this estimate of probability is not the objective truth? We know that. We know that from the fact that its called an estimate.
whitd (34bc65) — 7/1/2007 @ 9:30 pmYup. An estimate with no connection to reality.
Patterico (e7ff8a) — 7/1/2007 @ 10:05 pm“Yup. An estimate with no connection to reality.”
See, this is what i’m trying to correct. These idiotic total assertions. Its connected to reality as much as the assumption that judges are as accurate or more than juries. Its as connected to reality as the real data he uses on how judges and juries ruled. Its not unconnected to reality. Its data, plus assumptions (including proposing and going through various models) plus arithmetical manipulations. Its as connected to reality as those models and assumptions (i’m assuming you don’t find arithmetic unconnected to reality).
Anyway, it looks like you want to keep your narrow total view, and there’s no way to fix it. Who knows. Maybe god made you this stubborn.
whitd (e66d8d) — 7/2/2007 @ 8:10 amwhitd,
In addition to his conscious assumptions, on the reasonableness of which we can disagree, the author has completely failed to take into account the burden of proof and how it affects the relationship between procedural guilt and omniscient guilt. While he acknowledges its existence, nowhere can I find his estimate as to what the “reasonable doubt” standard means. As I’ve alluded previously, this is key, because if the reasonable doubt standard is fairly high, then it is absolutely not the case that anywhere near 10% of convicted defendants could be innocent by the omniscient standard, as he has claimed.
Naturally, when he assumes that judges are right more often than juries, and fudges the distinction between procedural guilt and omniscient guilt, he finds that a significant number of the juries’ guilty decisions must be mistakes. (Actually a higher percent even than the judges disagreed with, under the assumption, as I understand it, that if there is a given level of mistake found by the judge, there is probably a comparable level missed by the judge.)
DWPittelli (2e1b8e) — 7/2/2007 @ 6:27 pm“While he acknowledges its existence, nowhere can I find his estimate as to what the “reasonable doubt” standard means.”
He doesn’t need to. He relies on juries and judges to reach that decision.
whitd (57aa5a) — 7/2/2007 @ 6:53 pmwhitd,
DWPittelli (2e1b8e) — 7/2/2007 @ 7:08 pmNow who is being obtuse?