My father was a cop. Specifically, he was an arson investigator which meant that after a fire was put out by the fire department, he was called to determine if the fire was arson (meaning it was deliberately set) or accidental. Sometimes, a very young me would get to come along for this dirty business. “Dirty” because it’s messy, everything is wet and covered in soot.
Part of the investigation is to find the point of ignition. The method is very simple, look for the room that is the most destroyed, and then look for the “v.” The “v” is where the fire started (if there is more than one, it was much more likely arson), because fire burns up and out in its nearly sentient search for fuel. You can trace it right down to the single place where the fire started, and then begin looking for other clues like if there is a gasoline smell to the room, if the copper wiring has melted (copper melts at about 1000C, meaning that it was a pretty hot fire), or if the fire traveled in an odd manner.
I describe this because this information is objective. Sure “odd” is an impression that one might come to, but if you understand that fire burns up and instead the burn pattern has it snaking across the floor, you will say that it is odd.
In some cases (literally), there must be chemical analysis. Fire is a chemical reaction, and those reactions leave traces. Those traces will inform a person as to what burned and what, if anything, was used to “help” the fire along. Lab analysis in these cases is also objective, though I never witnessed any of these firsthand.
With every step in the crime scene investigation for this kind of crime an objective technique is used to reconstruct what happened. After that story is written (or more accurately re-written); the determination of arson or accident is made. There is very little room for subjective interpretation or bias to enter the case. The data is present—it just needs to be collected and arranged.
That whole story is to serve as a preface that we have all been taught, mostly through fictional portrayals, that this is how all criminal investigations work. We’ve seen the various techniques in so many different versions by different people that it’s hard to pin down a particularly jarring or incorrect show. The portrayal of forensic techniques in popular media is more like science fiction than reality; CSI’s green lasers are better at getting the writers out of the corners they wrote themselves into than the Doctor’s sonic screwdriver. This impression from years of film and television has served to remind us that committing a crime will get you caught. The impression here is that of a meticulous and scientific examination where the smallest particle of evidence is enough to find the perpetrator.
The impression has generated something known as the “CSI Effect.” There are two versions of this: the first is the one you may be the most familiar with—that prosecutors are often unable to get convictions because real forensic science isn’t as flashy as it is on the television show CSI, or that sometimes the evidence against the accused is a confession and an eyewitness not involving complex scientific testimony and CGI.
Studies into this effect though, have turned up empty. There was no finding that suspects were having their cases dismissed because a leggy blonde scientist wasn’t there to give a walk-through of blood splatter analysis. This CSI effect was no more than anecdotal complaints by frustrated prosecutors.
The second version is also anecdotal, but also more humorous, as the impression that these shows give leads perpetrators to try and coverup their crimes…but in doing so they provide more evidence to investigators.
Both descriptions should lead us to some heavy skepticism as to whether the effect is real though the latter one should give us more pause: we should never underestimate the effect pop-culture has on our perception of reality, i.e., a police officer does not have to tell you they are a police officer.
The list of forensic techniques at the disposal of the police includes, but is not limited to: fingerprint analysis, bullet analysis, object impressions, bite mark impressions, hair analysis, the polygraph testing, and DNA analysis. In this article let’s imagine that we have a gunshot murder (I do live in America after all), and that we are going to use each of these techniques to identify a suspect.
Let us begin our murder investigation with the discovery of a blood sample that has viable DNA on it, for the purpose of our fictional investigators this is great for the investigation. DNA currently sits as the gold standard in forensic investigations. It is the bar by which all the other standards are set. Most of my information is drawn from a report submitted to the American Justice Department by the National Academy of Sciences; where they performed a survey of studies on the forensic techniques used by law enforcement in the United States. I will note that despite the American focus of the report the problems described are not just American problems but cross the ocean to the UK as well.
The report sets DNA analysis as the epitome of a scientific forensic technique for a few reasons, to quote directly:
1] There are biological explanations for individual-specific findings; (2) the 13 STR loci used to compare DNA samples were selected so that the chance of two different people matching on all of them would be extremely small; (3) the probabilities of false positives have been explored and quantified in some settings (even if only approximately); (4) the laboratory procedures are well specified and subject to validation and proficiency testing; and (5) there are clear and repeatable standards for analysis, interpretation, and reporting.
One of the primary reasons that DNA can be relied upon is that the scientific principles which support it were hypothesized first and its application in forensic science followed it. This is not a case of a practical method which then found some post hoc justification later. What makes this feature interesting is that it is not how other techniques were developed. For example, there was never any science behind the polygraph machine (or lie detector) it was only justified because people felt that it could work. In some of the techniques I will discuss below, such as bullet striations, there is an impression that the technique can work but the conclusions that are derived from the method make much stronger claims than the method.
Page 133
DNA does not have this problem. Your DNA, minus some very odd cases, is your DNA. There is a connection between the sample and the person. Finding your DNA at a crime scene usually means that you were there. This connection brings us to (2) and (3), in the above quote, the methodology has been developed specifically to minimise false identification to a rare and quantified probability.
Importantly the last two factors generalise the science of DNA to the larger forensic world. For example, (4) claims that in order to be a DNA laboratory a facility must meet certain specifications. The person in the laboratory who conducts the tests must have a specific education and certification to be able to perform the test.
Finally, the fifth feature states what we assume is the case for all these tests: that the testing methods meet a scientific criterion of being repeatable. Two different people performing a DNA analysis on the same samples should reach the same conclusion. The problem, and what separates DNA testing from the following examples, is that this repeatability is not the case with other testing.
With DNA at the scene our murder suspect is going to have a difficulty in explaining how that happened. The report details several other techniques that are portrayed as reliable as DNA but have severe problems in meeting that threshold. Let’s roll a “chung chung” noise and return to our murder scene.
Our two detectives, one old and a recovering alcoholic (because this is the cliché) and his younger impetuous partner do not find DNA. They do have the bullet slug in the victim and a gun of matching caliber in the suspects’ flat. What follows is a scene so ubiquitous in police procedurals that it appears in everything from Law and Order to Law and Order UK. The pathologist has fired the gun into a tank full of ballistic gel and examined the slug. She seems reluctant to say it’s a match but the older detective explains that unless they get a match from the suspect’s gun they can’t make an arrest. The pathologist then offers some technobabble about striations and twist angles which the detectives use to get their arrest warrant. Then another “chung chung” and the crown is prepping their case.
The scene is as full of problems as it is clichéd. I’ll start with the science, “The fundamental problem with toolmark and firearms analysis is the lack of a precisely defined process (page 155).” A match involves the examiner concluding in which there is significant agreement between the slug pulled from the victim and the one from the recovered firearm. Yet the term “agreement” is ambiguous and changes between examiners. There is no definition of “agreement” to make a match. Further, even if there existed a 100% match, that only proves that the type of firearm was used in the crime, but not the token firearm held in official custody.
Guns aren’t made in a bespoken manner. They are mass produced. The barrel of the weapon, the thing which causes all the marks on the slug, are produced in long tubes and then cut for individual weapons. The slug can tell us the caliber of the weapon, which narrows the field; it can perhaps tell us the type of weapon if the number of grooves is particular to a type of gun. Generally, though, if we find out that the slug came out of a Glock-7 (the firearm of the UK army) then there are hundreds of thousands of possible matches save some deformity in the barrel. What our detectives have is a type of gun not a specific one. While ownership of the weapon by the suspect is a problem for the suspect it doesn’t prove they are the murderer.
Let us say that our detectives never found a firearm. Instead, they have a hair that does not match the victim but doesn’t have a follicle on it—so no DNA. Hair can tell us something, it can rule out a blonde if the hair is black. Shows like CSI give us the impression that a person can determine many more things about a hair, but none of that is borne out by the science. The report claims:
No scientifically accepted statistics exist about the frequency with which particular characteristics of hair are distributed in the population. There appear to be no uniform standards on the number of features on which hairs must agree before an examiner may declare a match
Page 160
Instead of a hair, our detectives find a clothing fiber. Not only do the same problems as before existing but there are more factors which can affect a fiber: sunlight exposure, cleaning, and wear. The only thing that can be determined by the examiner is that the fiber is artificial or natural, and sometimes what kind of fiber it is. Again, the portrayal of this in popular media is that a fiber match might as well be a video of the person committing the crime. The report is even more condemning of fiber analysis,
Fiber examiners agree, however, that none of these characteristics is suitable for individualizing fibers (associating a fiber from a crime scene with one, and only one, source) and that fiber evidence can be used only to associate a given fiber with a class of fibers.
Page 161
Let’s adjust our detective scenario one more time. Instead of submitting a hair or fiber, the detectives submit a fingerprint lifted from the scene and want a comparison with one taken from the suspect. You’re probably thinking that I’m some ACAB Antifa Anarchist who wants to discredit literally anything that the police do; surely fingerprint analysis is legitimate. We use fingerprints as an analogy for things that are unique.
Fingerprint analysis is properly called “Friction ridge analysis.” I want to get one thing out of the way immediately, I am not saying that fingerprints are not unique to the individual as the report admits,
some scientific evidence supports the presumption that friction ridge patterns are unique to each person and persist unchanged throughout a lifetime.
Page 143, 144
That much is true we can still use their uniqueness for analogies, but
Uniqueness and persistence are necessary conditions for friction ridge identification to be feasible, but those conditions do not imply that anyone can reliably discern whether two friction ridge impressions were made by the same person.
Page 144
At the risk of being repetitive, the issue is that no standardisation of the methods of comparison exist. It is up to the examiner whether they measure the swoops, the loops, or the number of ridges; between examiners though there is no agreement for standards. Scotland Yard may have a different standard than the FBI, but there is no good reason why the standardisation is different. If friction ridge analysis is a science in the same way that DNA is a science, there should be no differences in the analysis. The matches are determined through comparison of points, meaning that if a fingerprint has an “8 point” similarity with another one: that they share eight similar swoops or loops. France requires 12 points while the UK requires 16: this difference is nonsensical. The primary difference is that the UK police have a higher threshold for matching than France, which has a higher one than in the US (where the standard is up to the relative jurisdiction. Lowering the threshold only means that there is a higher likelihood of a false match which will lead to false arrests and accusations as was seen in the infamous arrest of Brandon Mayfield for the 2004 Madrid train bombings, where the US FBI was 100% positive that Mayfield’s print matched one from a piece of luggage found at the scene.
Consider the scenario: the fingerprint at the scene is on a drinking glass and it is getting compared with a print taken by the police. The police method of taking a fingerprint from a person is done in ideal conditions by someone trained to make the impression. They are comparing an ideal sample with a natural one. If you’ve ever fought with the fingerprint sensor on your phone/computer, you understand the problem. You set the sensor with a dead-on impression but when you unlock your mobile it’s not as flat and sometimes you have to hit it a few times.
Imagine that our older detective is sees a print on a high ball glass and calls for an examiner. There are several factors that can change the shape of a print. The material of the surface, the shape of the surface (i.e., if it is curved or not), the humidity, the temperature, etc. You can perform this at home: make a print on a piece of glass, then make another one a little firmer than that. They will not be significantly different, but they will be different. Now imagine that the print at the scene is a smudged partial on the rim of a highball glass and that partial has an eight-point match with the suspect. We don’t know if the match is because it was smudged or if it was a match regardless of the smudge. If we consider friction ridge analysis to be a science then the standards should be universal, the threshold should be the same across all jurisdictions.
The physical evidence aside, the other problem with that scene is the influence of the detectives on the examiner, this is a form of obvious bias. From a scientific perspective there exists no reason that the detectives are even discussing the situation with the examiner. The only purpose that serves is to bias the examiner into agreeing with the what the detectives, i.e., their co-worker, want. We would not tolerate this kind of influence, or even the appearance of it, in any other kind of scientific inquiry. This would be the same kind of interference as a CEO of a pharmaceutical company discussing the results of a clinical study with the people conducting the study.
For an honest examination the evidence should be submitted with only the absolute necessary information—in this case, two bullet slugs and then a chain of custody report. The examiner should not even be aware of what the case that samples refer to is. They shouldn’t know if it was from the famous murder or from the beer bottle someone throw at a referee. None of that information is relevant to whether the two samples match or not.
We as skeptics can claim what we do is important; that science, logic, and reason matter, that it is more than just explaining how a well-known psychic appears to be communicating to the dead, that facilitated communication has no scientific basis, or that homeopathy is literally nothing. Our skepticism is most important when people’s lives are on the line. In Democratic societies like ours, miscarriages of justice become the responsibility of the electorate. It is important to remember that when the crown, or the state, prosecutes a person suspected of a crime they are doing so on our behalf, and we bear a responsibility when innocent people go to jail.
