Continuing the (admittedly sporadic) reports on the AAAS meeting, I wanted to report on a great talk I went to about the Science behind Forensic Science. How accurate is it? I heard a great presentation by three speakers: Anne Marie-Mazza who helped write a report from the National Research Council on Forensic Science, Karen Kafadar, a Statistician from Indiana State, Greg Ridgeway, the Acting Director for the National Institute of Justice, which is the Science part of the Department of Justice (they also investigate things like body armor). I was excited before I even went to the talk because I had heard of the CSI effect in which juries essentially think that evidence should be like in those crime shows. I knew there were lots of inaccuracies on the forensics shows on tv. (For instance, why would people all work in the dark or green light? Turn on the light!) What I didn’t realize, however, was how little validation there was behind the techniques that I had taken for granted. Much of this was revealed in a 2009 report by the National Research Council, after a call by Congress for a study.
One of the first images that comes up when you type in Forensic Science is a fingerprint. It seems to be the bedrock of forensics. When I taught Forensics in my class, I told them all about the whorls, arches, and different ridge patterns and that all our fingerprints are unique…and that aside from DNA fingerprinting was extremely accurate. So it turns out I was wrong…Fingerprint evidence isn’t quite as reliable as we’ve all sort of believed (speaking for myself). It turns out there isn’t an official licensing or training program for fingerprint analysis. There aren’t published error rates, confidence levels, or general agreements on admissibility of evidence.
The NIJ commissioned a study to look at the reliability of top fingerprint analysts from the FBI and Spanish Police after a train bombing. The fingerprint evidence was confirmed by 3 independent analysts – perfect right? Except that these analysts weren’t blinded, and everyone tended to pick the indicated one. Fingerprints were randomly labeled “suitable” or “not suitable” for analysis and the 2nd examiner generally agreed with the statement for whichever fingerprint was labeled suitable. In other words, there was bias and influence from previous evaluators.
The recommendations for Fingerprint Analysis and Future Research?
- Blinded Evaluators
- Quantifying how Rare specific fingerprint features are in different populations
- Generating Confidence levels for predictions
- Determining the probabilistic measure of similarity of the crime scene fingerprint to an individual’s fingerprint
It turns out there are similar issues with blood splatter analysis, ballistics, handwriting analysis, etc. I’ve always accepted the idea we see on those forensics shows that each gun has a unique barrel pattern that is reflected on its bullets. It turns out that this isn’t necessarily supported by the evidence! The only type of forensic analysis that fared well in the report, as far as scientific validity, was DNA analysis. It was surprising to me that there hasn’t been as much work as I thought there had been on basic validation of the techniques that are reported on in a courtroom somewhere in the U.S. probably every day. The industry is changing, however. Instead of just the idea that “of course this evidence is good”, they are starting to develop statistics and quality standards, and may even develop an accreditation process for Forensic analysis. I think this is a good thing, and I maybe think that policy needs to follow to support this.
Some other interesting facts I picked up, and some good links:
- Apparently, not all states require medical training to be a medical examiner
- It costs lots of money and time to process the DNA. There are many unprocessed rape kits sitting in evidence that may eventually go bad.
- The Washington Post did a great investigative piece on some of the problems with forensic science. The videos on that site are especially interesting (and disturbing)
- The Innocence Project has helped to use DNA to free people who have been wrongly convicted based on faulty and unvalidated forensic evidence.
- Many forensics labs aren’t independent of the police force. The NRC Report and the NIJ both recommend that these labs separate themselves.
- While this isn’t related to the validity of the scientific evidence, many people are convicted based on eyewitness misidentification, and false confessions who are later exonerated by DNA evidence. The recent Ken Burns documentary on PBS, about the Central Park Five, was about these false confessions, and powerful. I think it might be easy for us to say we would never falsely confess if we haven’t been involved in an interrogation.
- Forensic Science has been around a long time. I heard this great piece on the Scientific American Podcast about early forensic science.