Debunking myths on genetics and DNA

Tuesday, December 20, 2011

Fingerprint evidence: not exactly what CSI showed you


Every scientific type of analysis has an error rate. I've mentioned it before: science is not about certainty, it's about accurately measuring the uncertainties. Unfortunately, when it comes to forensic sciences, this causes a logical problem: scientists like to talk about being 90% sure about something, but in trials there's only two outcomes: innocent or guilty. You can't do 90% guilty and 10% innocent.

It occurred to me that this was an issue when I heard somebody talk about how fingerprint analysis has no error rate. Being a statistician, all sorts of red flags rose in my head. Of course there's an error rate! Any procedure has an error rate because the error rate is a definition: you count the number of successful identifications and divide by the total number of comparisons made. Maybe the fingerprint matching has never failed? Unfortunately, that's not true

As some of you know, I was in Los Angeles last month and I was lucky enough to speak to a fingerprint analyst with the LAPD. She explained the procedure to me, which is pretty cool, actually. Blood fingerprints are photographed. Everything else is dusted: black powder is used for clear surfaces, and white powder for dark surfaces. The powder sticks to the oily residue from hands and the rest is brushed away. The analyst then applies tape to the surface, which lifts the dust that has attached to the oily residues. This accurately reproduces the print onto the tape and the tape is then transferred to a card.

The typical procedure is to scan the card into the computer and send it to a number of databases, the largest of which is IAFIS, maintained by the FBI. An automated algorithm comes back with 2-3 possible matches with the prints lifted from the scene, and at that point a person compares the matches from the database with the in order to make a final identification. If the database comes back with no match, but the police has obtained the prints from a possible suspect, then again the comparison is done by hand.

Just like any other procedure that deems itself as "scientific," fingerprint analysis too needs an error rate, and I'm not alone in making this claim: in 2010 UCLA law professor Jennifer Mnookin obtained a federal grant in order to study error rates in fingerprint evidence. The difficulties are intrinsic: you have to fold in the variability from analyst to analyst and from case to case, as some fingerprints are lifted in ideal situations whereas others are only partial, blurred, etc.

To explain the importance of this type of research, in a 2010 paper [1] Mnookin asks the following question to two hypothetical law students:
"Is fingerprint identification one of the most secure forms of evidence we have, or is its scientific validity remarkably untested?"
Sadly, one of her hypothetical students
"would discover that the scientific validity of fingerprint evidence is surprisingly untested. He would ascertain, for example, that there were no generally shared and validated objective standards for declaring a ‘match’ either required by judges within the courtroom as a precondition to admissibility, or self-imposed by the fingerprint community—that in the end, deciding whether or not a match existed was left to the discretion of each examiner based on his judgement and experience. He would find, in sharp contrast to, say, DNA profiling, that there was no fully specified statistical model of fingerprinting, that would permit the likelihood that two prints came from the same person to be expressed in probabilistic terms."
Mnookin concludes:
"Assuming that errors were not randomly distributed across the array of test prints, proficiency tests designed genuinely to test both the method and its limits might provide useful information about the circumstances that tended to lead to errors, and perhaps could lead to practices that could reduce the likelihood of making such mistakes. Such proficiency tests are technically feasible, and whatever they found would teach us a good deal more about the frequency of errors than we know at present."
To get a more rounded perspective on the matter, I contacted my friend Mark Pryor, an assistant district attorney with the County of Travis, in Texas. Mark was very happy to hear from me, and he was even happier to tell me that he had just found a home for his novel The Bookseller, signing a three-book-deal with Seventh Street Books, a new mystery and thriller imprint from Prometheus Books. Way to go, Mark! He answered my questions while floating twelve inches above ground...

EEG: I understand that every time you have an expert witness take the stand they have to state what their expertise is, and their background and how many cases they've investigated, etc. Are they ever asked, over the course of the testimony, what the error rate for their kind of analysis is? Is it at all required for some kind of analyses?

MP: Yes, they always explain their qualifications, experience and training. This is for two reasons: one, to qualify them as an expert, so that they pass what's known as the Daubert test for scientific expert testimony. That's a call for the judge. But it's also so the jury can know them a little, understand who this person is giving the scientific opinion/testimony. I have never heard one asked what about error rates, actually.

EEG: Has a defense lawyer ever questioned fingerprint analyses during one of your cases, and if so, how did you reply to that? If not, how would you reply to that?

MP: I have not taken a fingerprint case to trial yet, so no. I'm sure if I did, defense counsel would thoroughly question the expert, maybe about error rates, certainly about the procedures they use and the practices in place to make sure there were no mistakes. I think what I would emphasize in my re-direct of such an expert is precisely those same things: making it clear neither I nor the fingerprint expert has any interest in making a mistake and blaming the wrong person for a crime. In other words, I want my print expert to come across precisely as he is: a disinterested expert on the science of fingerprint analysis.

EEG: More in general, do you ever "question" the scientific evidence that detectives bring to your desk before deciding whether or not to issue an arrest warrant?

MP: I personally don't have anything to do with arrest warrants, a detective will get that from the judge directly. Sometimes I'll review one if it's a big case, but I generally won't see any scientific evidence - usually because the arrest is based on non-scientific evidence because a lot of the science stuff takes a while to process (I'm thinking of DNA here). Now, for trial I always look very hard at ALL my evidence, scientific included. Again, for two reasons: I don't want to try an innocent person, and also because I don't want to find out any mistakes/weaknesses once I'm in trial, I want to know those in advance!

I confess I'm still a bit troubled by the process. TV shows like CSI make it look like scientific evidence is either black or white, while unfortunately there's a full range of grays in between. And like Mark said, the bottom line is not to try an innocent person. I'm glad the law is represented by scrupulous people like Mark, but I'm also grateful that Professor Mnookin got her grant. I couldn't find any update on her website, but I'll keep an eye on it because I'm eager to find out about their results.

As for Mark, his book The Bookseller will hit the shelves towards the end of 2012, but in the meantime you can follow his blog, DA Confidential, where he discusses being an assistant district attorney, a father, a husband, and a writer.

[1] Mnookin, J. (2007). The validity of latent fingerprint identification: confessions of a fingerprinting moderate Law, Probability and Risk, 7 (2), 127-141 DOI: 10.1093/lpr/mgm022

7 comments:

  1. I'll be interested in how that study turns out also. I recall seeing a show sometime last year (I don't recall the title) that talked about fingerprint evidence and how the analysis is much more of an art form than a science. Accuracy can apparently vary pretty widely between two different analysts.
    The show mentioned some pretty different results from a set of sample fingerprints to "matches."

    ReplyDelete
  2. Luckily there's so many other things that contribute to a conviction, but still, it does sound a bit troubling... I know those guys are trained in what they do, but we're all human and mistakes happen. Also, sometimes you don't even have full prints, just smears.

    ReplyDelete
  3. Surely it would be quite easy to take the human out of the loop and get the error rate of the IAFIS matching algorithm.

    ReplyDelete
  4. Thanks for writing this. I'm wowed by how little information about finger print testing accuracy exists. You learn a new thing every day, and sometimes that thing makes you feel a bit uncomfortable...

    I found this paper [1] which reviews what is currently known about sensitivity. It's pretty hefty, and appears to carry a strong POV. But the information in it is well worth reading.

    Learned while searching for it that Forensics journals are way cooler than normal science ones cos they use gory photos to break the photo up where we'd just use graphs. They also seem way more rhetorical.

    It points out a 50% fail rate in the FBI internal exam, and a 5% error rate in regular lab accuracy exams.
    But, it points out that the FBI test is sat by a self selection of 'the best' examiners, but is deliberately hard, and that the lab accuracy test is by post and may therefore be a consensus of all the examiners in the lab.
    So er, inconclusive.

    It makes finger print analysis look a bit like alternative medicine - great claims, great confidence, and a legitimate looking professional certification system, but no real evidence that it works.
    Until someone refutes energy healing, you could say the same for that.
    I'm sure Fingerprinting would stand up, and a lot of court evidence is hard to quantify. That's what juries are for. But it sounds like they are being mislead here.


    [1]http://www.nasams.org/forensics/for_lib/Documents/1128572507.37/FinalProof.pdf

    ReplyDelete
  5. @Anonymous: no, because by law IAFIS matches need to be reviewed by a person and that person will be the one testifying in court.

    @Sam: Thanks, I'll check out the paper.

    ReplyDelete
  6. http://www.miller-mccune.com/legal-affairs/bias-and-the-big-fingerprint-dust-up-3629/

    I would also suggest you read about contextual bias in fingerprint analysis

    ReplyDelete
  7. I was taken aback during the O. J. trial when the head of Cellmark labs, which did the DNA analysis, admitted on the stand that Cellmark had stopped testing their own accuracy by doing analyses sent to them blind for that purpose. IIRC, at that time the California State labs had about a 2% error rate. It also seems that fingerprint examiners are not sent prints for the purpose of determining an error rate. Shouldn't that be SOP?

    ReplyDelete

Comments are moderated. Comments with spam links will be deleted and never published. So, if your intention is to leave a comment just to post a bogus link, please spare your time and mine. To all others: thank you for leaving a comment, I will respond as soon as possible.