Fingerprint evidence: not exactly what CSI showed you

Every scientific type of analysis has an error rate. I've mentioned it before: science is not about certainty, it's about accurately measuring the uncertainties. Unfortunately, when it comes to forensic sciences, this causes a logical problem: scientists like to talk about being 90% sure about something, but in trials there's only two outcomes: innocent or guilty. You can't do 90% guilty and 10% innocent.

It occurred to me that this was an issue when I heard somebody talk about how fingerprint analysis has no error rate. Being a statistician, all sorts of red flags rose in my head. Of course there's an error rate! Any procedure has an error rate because the error rate is a definition: you count the number of successful identifications and divide by the total number of comparisons made. Maybe the fingerprint matching has never failed? Unfortunately, that's not true. 

As some of you know, I was in Los Angeles last month and I was lucky enough to speak to a fingerprint analyst with the LAPD. She explained the procedure to me, which is pretty cool, actually. Blood fingerprints are photographed. Everything else is dusted: black powder is used for clear surfaces, and white powder for dark surfaces. The powder sticks to the oily residue from hands and the rest is brushed away. The analyst then applies tape to the surface, which lifts the dust that has attached to the oily residues. This accurately reproduces the print onto the tape and the tape is then transferred to a card.

The typical procedure is to scan the card into the computer and send it to a number of databases, the largest of which is IAFIS, maintained by the FBI. An automated algorithm comes back with 2-3 possible matches with the prints lifted from the scene, and at that point a person compares the matches from the database with the in order to make a final identification. If the database comes back with no match, but the police has obtained the prints from a possible suspect, then again the comparison is done by hand.

Just like any other procedure that deems itself as "scientific," fingerprint analysis too needs an error rate, and I'm not alone in making this claim: in 2010 UCLA law professor Jennifer Mnookin obtained a federal grant in order to study error rates in fingerprint evidence. The difficulties are intrinsic: you have to fold in the variability from analyst to analyst and from case to case, as some fingerprints are lifted in ideal situations whereas others are only partial, blurred, etc.

To explain the importance of this type of research, in a 2010 paper [1] Mnookin asks the following question to two hypothetical law students:

"Is fingerprint identification one of the most secure forms of evidence we have, or is its scientific validity remarkably untested?"

Sadly, one of her hypothetical students

"would discover that the scientific validity of fingerprint evidence is surprisingly untested. He would ascertain, for example, that there were no generally shared and validated objective standards for declaring a ‘match’ either required by judges within the courtroom as a precondition to admissibility, or self-imposed by the fingerprint community—that in the end, deciding whether or not a match existed was left to the discretion of each examiner based on his judgement and experience. He would find, in sharp contrast to, say, DNA profiling, that there was no fully specified statistical model of fingerprinting, that would permit the likelihood that two prints came from the same person to be expressed in probabilistic terms."

Mnookin concludes:

"Assuming that errors were not randomly distributed across the array of test prints, proficiency tests designed genuinely to test both the method and its limits might provide useful information about the circumstances that tended to lead to errors, and perhaps could lead to practices that could reduce the likelihood of making such mistakes. Such proficiency tests are technically feasible, and whatever they found would teach us a good deal more about the frequency of errors than we know at present."

To get a more rounded perspective on the matter, I contacted my friend Mark Pryor, an assistant district attorney with the County of Travis, in Texas. Mark was very happy to hear from me, and he was even happier to tell me that he had just found a home for his novel The Bookseller, signing a three-book-deal with Seventh Street Books, a new mystery and thriller imprint from Prometheus Books. Way to go, Mark! He answered my questions while floating twelve inches above ground...

EEG: I understand that every time you have an expert witness take the stand they have to state what their expertise is, and their background and how many cases they've investigated, etc. Are they ever asked, over the course of the testimony, what the error rate for their kind of analysis is? Is it at all required for some kind of analyses?

MP: Yes, they always explain their qualifications, experience and training. This is for two reasons: one, to qualify them as an expert, so that they pass what's known as the Daubert test for scientific expert testimony. That's a call for the judge. But it's also so the jury can know them a little, understand who this person is giving the scientific opinion/testimony. I have never heard one asked what about error rates, actually.

EEG: Has a defense lawyer ever questioned fingerprint analyses during one of your cases, and if so, how did you reply to that? If not, how would you reply to that?

MP: I have not taken a fingerprint case to trial yet, so no. I'm sure if I did, defense counsel would thoroughly question the expert, maybe about error rates, certainly about the procedures they use and the practices in place to make sure there were no mistakes. I think what I would emphasize in my re-direct of such an expert is precisely those same things: making it clear neither I nor the fingerprint expert has any interest in making a mistake and blaming the wrong person for a crime. In other words, I want my print expert to come across precisely as he is: a disinterested expert on the science of fingerprint analysis.

EEG: More in general, do you ever "question" the scientific evidence that detectives bring to your desk before deciding whether or not to issue an arrest warrant?

MP: I personally don't have anything to do with arrest warrants, a detective will get that from the judge directly. Sometimes I'll review one if it's a big case, but I generally won't see any scientific evidence - usually because the arrest is based on non-scientific evidence because a lot of the science stuff takes a while to process (I'm thinking of DNA here). Now, for trial I always look very hard at ALL my evidence, scientific included. Again, for two reasons: I don't want to try an innocent person, and also because I don't want to find out any mistakes/weaknesses once I'm in trial, I want to know those in advance!

I confess I'm still a bit troubled by the process. TV shows like CSI make it look like scientific evidence is either black or white, while unfortunately there's a full range of grays in between. And like Mark said, the bottom line is not to try an innocent person. I'm glad the law is represented by scrupulous people like Mark, but I'm also grateful that Professor Mnookin got her grant. I couldn't find any update on her website, but I'll keep an eye on it because I'm eager to find out about their results.

As for Mark, his book The Bookseller will hit the shelves towards the end of 2012, but in the meantime you can follow his blog, DA Confidential, where he discusses being an assistant district attorney, a father, a husband, and a writer.

[1] Mnookin, J. (2007). The validity of latent fingerprint identification: confessions of a fingerprinting moderate Law, Probability and Risk, 7 (2), 127-141 DOI: 10.1093/lpr/mgm022