The L.A. Times has an interesting article about the application of probability measures to “cold hit” cases made from DNA databases. I find the statistical arguments made in the article to be unconvincing, but due to my lack of training in this area, I remain completely humble about my ability to properly analyze the issue. However, experts have widely divergent opinions on the matter — a fact you’d never learn reading the article.
The article begins by describing a 1970s rape/murder scene. A match was made from badly deteriorated DNA that bore only 5 1/2 of the possible 13 markers available. When all 13 markers are available for a match, the probability of a random person bearing the same profile can run to 1 in a quadrillion — thousands of times the number of people on the planet. Because of the lack of the 13 markers in this case, the chance was lowered to 1 in 1.1 million.
This is known as a “random match probability” and the article describes it as the “rarity of a particular DNA profile in the general population.”
At Puckett’s trial earlier this year, the prosecutor told the jury that the chance of such a coincidence was 1 in 1.1 million.
Jurors were not told, however, the statistic that leading scientists consider the most significant: the probability that the database search had hit upon an innocent person.
In Puckett’s case, it was 1 in 3.
The article restates the proposition again later in the article:
In every cold hit case, the panels advised, police and prosecutors should multiply the Random Match Probability (1 in 1.1 million in Puckett’s case) by the number of profiles in the database (338,000). That’s the same as dividing 1.1 million by 338,000.
For Puckett, the result was dramatic: a 1-in-3 chance that the search would link an innocent person to the crime.
It seems to me that the conclusion does not logically follow at all. The formulation simply can’t be right. The suggestion appears to be that the larger the database, the greater the chance is that the hit you receive will be a hit to an innocent person. I think that the larger the database, the greater the probability of getting a hit. Then, once you have the hit, the question becomes: how likely is it that the hit is just a coincidence?
An example makes it simpler.
Let’s say the random match probability for a DNA profile is one in 13.4 billion. In such a case, it seems very unlikely that the hit you get will come back to a different person than the person who left the DNA at the crime scene. Now assume that your database contains all 6.7 billion people on the planet. It’s virtually certain that you will get a hit, of course. But if you got a hit — only one hit — you would intuitively feel certain that you had the right person from that hit.
Yet the logic of the article would seem to say you take 13.4 billion and divide it by the size of the database (6.7 billion). making a 1-in-2 chance (50%) that you have the wrong person (an “innocent person”).
I say hogwash. And I think my example shows why it’s confusing and potentially misleading to use the word “innocent” in these calculations.
My off-the-cuff reaction — and keep in mind, I have no experience in statistics — is that the people who advocate this approach are measuring the question:
1. What are the chances that a search of this database will turn up a match with the DNA profile?
when the truly relevant question is, instead:
2. What are the chances that any one person whose DNA matches a DNA profile is indeed the person who left the DNA from which the profile is taken?
There is a third, rather silly question whose answer seems obvious, but which I will raise for the purposes of relating to an analogy I will make:
3. Once a match has been made through the database, what is the chance that the person whose DNA provided the match will match the DNA profile?
This last one is obviously almost 100%, the lack of complete certainty owing purely to human error; taking human error out of the equation for a theoretical analysis, it’s a tautology: a match is a match.
It seems to me that this is a useful analogy: everyone knows a coin has a 50/50 chance of coming up heads. If I give you a room that has 10,000 coins that were randomly tossed in the air and have landed on the ground, the chances that at least one of those coins landed heads are very nearly approaching 100% certainty (question 1). But the chances that any one of those coins was going to come up heads before it was tossed is still 50% (question 2).
Now, if I tell you to go find me a coin that has come up heads, then the chances it did come up heads are (absent human error) 100% (question 3). But, the chances that it was going to come up heads before it was tossed are still 50% . . . and always will be, no matter how many coins are in the room. You’re almost certain to find one with heads in a room with a larger database (thousands of coins), but the chances that it was going to come up heads always remain the same.
Applying the analogy to a DNA database, it seems to me that the size of the database increases your chances of a hit. But the chances that the profile obtained from your hit is a coincidence will always remain the same, and will always be a function of the number of loci and their frequency in the relevant populations.
The L.A. Times article makes it sound as though it’s quite well accepted that jurors are constantly being misled:
Jurors are often told that the odds of a coincidental match are hundreds of thousands of times more remote than they actually are, according to a review of scientific literature and interviews with leading authorities in the field.
. . . .
[B]ecause database searches involve hundreds of thousands or millions of comparisons, experts say using the general-population statistic can be misleading.
The closest you get to an acknowledgement that not everybody agrees is a passing reference to the fact that this assertion “has been widely but not universally embraced by scientists.”
“Not universally” is quite the understatement. Apparently, there is a debate raging about this among statisticians. Law professor David H. Kaye explains that, while many agree with the analysis described in the L.A. Times article, there is a theory out there that the use of the database “actually increases the probative value of the match.” (I have an e-mail in to Professor Kaye to ask him for further comment.)
The argument to which Professor Kaye refers was made in a Michigan Law Review article by Peter Donnelly, Professor of Statistical Science and Head of the Department of Statistics at the University of Oxford, and Richard D. Friedman, a law professor at the University of Michigan. The first page of their law review article is here. An earlier version of the argument was apparently made by Donnelly with David Balding in a paper titled “Evaluating DNA Profile Evidence When the Suspect is Identified Through a Database Search,” according to mathematician Keith Devlin of Stanford.
Devlin appears to agree with the approach described in the L.A. Times article. However, he says:
Personally, I (together with the collective opinion of the NRC II committee) find it hard to accept Donnelly’s argument, but his view does seem to establish quite clearly that the relevant scientific community (in this case statisticians) have not yet reached consensus on how best to compute the reliability metric for a cold hit.
You’d never know that reading the L.A. Times article, which implies that all but the most rabid pro-law enforcement shills agree that jurors are being given bogus statistics.
[UPDATE: For proof as to how conclusively the paper portrays this point of view, look at this image of what appears on the front page of today's Sunday paper:
Tell me where in that image you see any hint that "the relevant scientific community (in this case statisticians) have not yet reached consensus" as mathematician Devlin states.]
The paper wraps up the article by suggesting that the real probability of a coincidence is not 1 in 1.1 million, but 1 in 3:
In the end, however, jurors said they found the 1-in-1.1-million general-population statistic Merin had emphasized to have been the most “credible” and “conservative.” It was what allowed them to reach a unanimous verdict.
“I don’t think we’d be here if it wasn’t for the DNA,” said Joe Deluca, a 35-year-old martial arts instructor.
Asked whether the jury might have reached a different verdict if it had been given the 1-in-3 number, Deluca didn’t hesitate.
“Of course it would have changed things,” he said. “It would have changed a lot of things.”
By the way, in the case described in the L.A. Times article, there was more than just the cold hit. In addition to the fact that the defendant was a serial rapist who described his rapes as “making love” — the same terminology used by the murderer — the prosecution also showed the following:
[Defendant] Puckett “happened to be in San Francisco in 1972,” Merin told jurors in his opening argument. Merin could not place Puckett in [victim] Sylvester’s neighborhood on the day of the slaying. But Puckett had applied for a job near the medical center where Sylvester worked.
With the court lights dimmed and a photo of Sylvester’s naked body displayed on a screen, Merin argued that Puckett’s 1977 sexual assaults showed an “MO” consistent with Sylvester’s killing.
In each of those crimes, Puckett had posed as a police officer to gain the woman’s trust. The absence of forced entry to Sylvester’s apartment indicated her killer had also used a ruse, Merin said.
Puckett had kidnapped his victims by holding a knife or ice pick to their necks, leaving scratches similar to those found on Sylvester’s neck — what Merin called “his signature.”
I now throw open the matter for discussion.
UPDATE: Radley Balko has posted on this. He agrees with the L.A. Times experts. I have posted some counterarguments in his comments.
UPDATE x2: Follow-up post here with helpful responses from Prof. Kaye.
UPDATE x3: Statistics always opens the possibility of using language that doesn’t describe what’s really going on. For example, in this post I referred to “random match probability” as “in essence, the chance that two unrelated people will share the same genetic markers.” I’m not comfortable that this is right, and have removed the sentence. Random match probability refers to the expected frequency of a set of markers appearing in a population of unrelated individuals. I think it’s best to stick with that definition.