8/22/2019 Can Bad Science Be Good Evidence
1/40Electronic copy available at: http://ssrn.com/abstract=1448744
University of Virginia School of LawPublic Law and Legal Theory Research Paper Series No. 2009-14
Can Bad Science Be Good Evidence? Lie Detection,
Neuroscience and the Mistaken Conflation of Legal and
Scientific Norms
Frederick SchauerUniversity of Virginia School of Law
October 2009
This paper may be downloaded without charge from the
Social Science Research Network Electronic Paper Collection:
http://ssrn.com/abstract=1448744
A complete index of University of Virginia School of Law research papers is available at
Law and Economics:http://www.ssrn.com/link/U-Virginia-LEC.html
Public Law and Legal Theory:http://www.ssrn.com/link/U-Virginia-PUB.html
http://ssrn.com/abstract=1448744http://ssrn.com/abstract=1448744http://www.ssrn.com/link/U-Virginia-LEC.htmlhttp://www.ssrn.com/link/U-Virginia-LEC.htmlhttp://www.ssrn.com/link/U-Virginia-LEC.htmlhttp://www.ssrn.com/link/U-Virginia-PUB.htmlhttp://www.ssrn.com/link/U-Virginia-PUB.htmlhttp://www.ssrn.com/link/U-Virginia-PUB.htmlhttp://www.ssrn.com/link/U-Virginia-PUB.htmlhttp://www.ssrn.com/link/U-Virginia-LEC.htmlhttp://ssrn.com/abstract=14487448/22/2019 Can Bad Science Be Good Evidence
2/40Electronic copy available at: http://ssrn.com/abstract=1448744
Can Bad Science Be Good Evidence? Lie Detection,
Neuroscience and the Mistaken Conflation of Legal and
Scientific Norms
Frederick Schauer
Abstract
As the capabilities of cognitive neuroscience, in particular functional magnetic resonance
imaging (fMRI) 'brain scans,' have become more advanced, some have claimed that fMRI-based
lie-detection can and should be used at trials and for other forensic purposes to determine
whether witnesses and others are telling the truth. Although some neuroscientists have
promoted such claims, most aggressively resist them, and arguing that the research on
neuroscience-based lie-detection is deeply flawed in numerous ways. And so these
neuroscientists have resisted any attempt to use such methods in litigation, insisting that poor
science has no place in the law. But although the existing studies have serious problems of
validity when measured by the standards of science, and true as well that the reliability of such
methods is significantly lower than their advocates claim, it is nevertheless an error to assume
that the distinction between good and bad science, whether as a matter of validity or of
reliability, is dispositive for law. Law is not only about putting criminals in jail, and numerous
uses of evidence in various contexts in the legal system require a degree of probative value far
short of proof beyond a reasonable doubt. And because legal and scientific norms, standards,
and goals are different, good science may still not be good enough for some legal purposes,
and, conversely, some examples of bad science my, in some contexts, still be good enough for
law. Indeed, the exclusion of substandard science, when measured by scientific standards, may
have the perverse effect of lowering the accuracy and rigor of legal fact-finding, because the
exclusion of flawed science will only increase the importance of the even more flawed non-
science that now dominates legal fact-finding. And thus the example of neuroscience-based lie
detection, while timely and important in its own right, is even more valuable as a case study
suggesting that Daubert v. Merrill-Dow Pharmaceuticals may have sent the legal system down a
false path. By inappropriately importing scientific standards into legal decision-making with
little modification, Daubert confused the goals of science with those of law, a mistake that it is
not too late for the courts to correct.
8/22/2019 Can Bad Science Be Good Evidence
3/40Electronic copy available at: http://ssrn.com/abstract=1448744
1
Forthcoming in Cornell Law Review, vol. 95 (2010), draft of 09/08/2009
CAN BAD SCIENCE BE GOOD EVIDENCE? NEUROSCIENCE, LIE-DETECTION, AND BEYOND
Frederick Schauer1
I. INTRODUCTION
How should the legal system confront the advances in the brain sciences that may
possibly allow more accurate determinations of veracity lie detecting than those that now
pervade the litigation process? In this Essay I seek to question the view, widespread among
the scientists most familiar with these advances, that the neuroscience of lie-detecting is not,
or at least not yet, nearly reliable enough to be used in civil and criminal litigation or for related
forensic purposes. But in challenging the neuroscientists and their allies, I make no claims for
the science of lie-detecting that go beyond the current state of scientific knowledge or, more
importantly, my own ability to speak about the relevant scientific developments. Rather, I
1David and Mary Harrison Distinguished Professor of Law, University of Virginia. This Essay was
presented on September 5, 2009, at the Mini-Foro on Proof and Truth in the Law, Institute for
Philosophical Research, Universidad Nacional Autonoma de Mxico (UNAM), on April 7, 2009,
as an Inaugural Lecture at the University of Virginia School of Law, and on June 1, 2009, at theDuck Conference on Social Cognition. Many of the ideas presented here were generated during
various meetings of the John D. and Catherine T. MacArthur Foundations Law and
Neuroscience Project, whose tangible and intangible support I am delighted to acknowledge.
Detailed and constructive comments by Charles Barzun, Teneille Brown, Greg Mitchell, John
Monahan, and Bobbie Spellman have made this version immeasurably better than its
predecessors.
8/22/2019 Can Bad Science Be Good Evidence
4/40
2
argue that because laws goals and norms are different from those of science, there is no more
reason to impose the standards of science on law than there is to impose the standards of law
on science. Law must and should use science, and should always prefer good science to bad,
but in some contexts good science may still not be good enough for law, and in other contexts
hence the title of this Essay bad science, if measured by the standards of scientists, may still
have valuable legal uses. To be clear, however, my goal in this paper is decidedly notto argue
that neuroscience-based lie detection should, now or even in the foreseeable future,
necessarily be admissible in court or legitimately used for other forensic purposes. Rather, it is
to argue that the question whether neuroscience-based lie detection should be used for legal-
system purposes is a question that cannot be answered exclusively according to scientific
standards of reliability and validity. Science can (and should) inform the legal system about the
facts, including facts about degrees of reliability and the extent of experimental validity, but the
ultimate normative and institutional question of whether and when, if at all, a given degree of
validity or reliability is sufficient for this or that legal or forensic purpose is a legal and not a
scientific question.
In important respects the analysis of the potential legal uses of neuroscience-based lie-
detection is more case study than discrete topic. Most of what I argue here applies to other
forms of lie-detection, to other forms of scientific evidence, and indeed to evidence generally.
And as I elaborate in the latter part of this Essay, my central theme may call into doubt some
dimensions of the modern revolution in the standards for the admission of scientific evidence.
8/22/2019 Can Bad Science Be Good Evidence
5/40
3
Starting with Daubert v. Merrill Dow Pharmaceuticals, Inc.,2and continuing through General
Electric Co. v. Joiner3and Kumho Tire Co., Ltd. v. Carmichael,
4the Supreme Court over the past
sixteen years has attempted to deal with the very real problem of junk science by imposing
increasingly stringent scientific standards of reliability and experimental validity on the
admissibility of scientific evidence and expert testimony in the federal courts. In dealing with
science and experts but not with the myths and superstitions that pervade the fact-finding
process, however, the Court may, unintentionally have lowered the quality of evidence
generally. By discouraging poor science while leaving non-science untouched, the Daubert
revolution may perversely have increased reliance on the even worse science that dominates
the litigation process by not masquerading as science at all. There may not be an easy solution
to this problem, but its identification suggests that Daubertmay have created as many
problems as it solved. The revolution in scientific and expert testimony that started with
Daubert, therefore, is, or at least should be, far from over.
II. NEUROSCIENCE-BASED LIE-DETECTION: CLAIMS AND COUNTER-CLAIMSI commence by describing the current controversy over the legal and forensic
5uses of
neuroscience-based lie-detection. In some respects the controversy should come as no
surprise. The common law litigation process places huge reliance on the sworn testimony of
2509 U.S. 579 (1993).
3522 U.S. 136 (1997).
4526 U.S. 137 (1999).
5In this Essay I use forensic, as distinguished from legal, to refer to those dimensions of
criminal investigation that precede or exist apart from the trial
8/22/2019 Can Bad Science Be Good Evidence
6/40
4
witnesses, a phenomenon that is itself worthy of note.6
Many other methods of factual
investigation, after all, employ dramatically different approaches relying far more heavily on
primary rather than secondary sources of knowledge. The scientist who seeks to determine
whether drinking red wine reduces the likelihood of heart disease does not summon
representatives of the wine industry and the Temperance League to each make their cases,
following which she decides which of the two advocates is more believable. Rather, she
engages in the kind of primary research we call experimentation. So too with historians when
they do original archival research, psychologists when they conduct experiments on subjects,
empirical economists when they perform multiple regressions with large data sets,
oceanographers when the explore the sea with scientific instruments or submersible
watercraft, and researchers for policy-makers when they combine techniques such as these and
others to determine the factual terrain for which they will make policy.7
6Although sworn witness testimony also plays a large role in the civil law, the qualification in
the text to the common law is a function of the somewhat larger role that judges play in many
civil law countries in managing the process of direct factual investigation, especially in criminal
cases. See Mireille Delmas-Marty & J.R. Spencer, European Criminal Procedures (2002);
Jacqueline Hodgson, French Criminal Justice: A Comparative Account of the Investigation and
Prosecution of Crime in France (2005).
7Testimony and other forms of indirect evidence do play a significant role outside of law. See,
for example, C.A.J. Coady, Testimony: A Philosophical Study (1992); Elizabeth Fricker,
Testimony: Knowing Through Being Told, in Handbook of Epistemology 109 (I. Niiniluoto, M.
Sintonen, & J. Wolenski, eds., 2004); Axel Gelfert, Indefensible Middle Ground for LocalReductionism about Testimony, 22 Ratio (new series) 170 (2009); John Hardwig, The Role of
Trust in Knowledge, 88 J. Phil. 693 (1991). Nevertheless, law is noteworthy in relying on
testimony and authority more than most other disciplines, and, conversely, and especially in
courtroom settings, relying substantially less on direct investigation and experimentation.
*A+uthority and hierarchy play a role in law that would be inimical to scientific investigation.
Richard A. Posner, The Problems of Jurisprudence 62 (1990).
8/22/2019 Can Bad Science Be Good Evidence
7/40
5
Once we grasp the diverse array of primary techniques for determining facts -- for
figuring out what is or was the case we can understand how unusual the legal system is in
routinely using party-generated witnesses to provide information as to which they, but not the
trier of fact, have first-hand knowledge,8
and as to which the trier of fact is ordinarily precluded
from obtaining the first-hand knowledge that in other domains remains the gold standard for
empirical reliability. Still, the legal system we have, idiosyncratic as it is within the realm of
empirical inquiry in relying so heavily on second-hand knowledge, is one in which it is often
important to determine which of two opposing witnesses is telling the truth.
Of course not all of litigation involves a conflict between a truth-teller and a liar. Honest
misperceptions and more-or-less honest omissions, exaggerations, shadings, fudgings,
slantings, bendings, and hedgings9
are an omnipresent feature of modern litigation. But so too
is flat-out lying, and it should come as no surprise that because the legal system relies far more
heavily on the reports of witnesses than on primary investigation by the trier of fact, the law
has long been preoccupied with trying to assess whether the witnesses who testify in court (or
otherwise provide information for legal or forensic decision-making) are telling the truth.
Historically, the law relied on the oath to serve the truth-warranting function. When
people genuinely believed that lying under oath would send them to hell, the law could
8Fed. R. Evid. 609.
9See Frederick Schauer & Richard Zeckhauser, Paltering, in Deception: From Ancient Empires to
Internet Dating 38 (Brooke Harrington, ed., 2009).
8/22/2019 Can Bad Science Be Good Evidence
8/40
6
comfortably rely on a witnesss fear of burning for eternity to provide confidence that
witnesses were unlikely to say things they believed to be untrue.10
As religious belief diminished, or at least as laws confidence in it as a guarantor of truth
waned, the legal system had its own substitute, increasingly relying on faith in the lie-exposing
powers of vigorous cross-examination. As celebrated in the Perry Mason television series of the
1950s and 1960s, and as reinforced by numerous items of popular culture since, the legal
system has long believed that cross-examination will so reduce the effectiveness of lying that a
truth-determining system relying on witness testimony and subsequent cross-examination
would not be unacceptably vulnerable to intentional deception.11
More importantly, and
because cross-examination is far less effective in exposing lies and liars than television writers
and their viewers believe, the legal system has placed its faith in judges and juries. Now, the
task of determining veracity, and credibility in general, has been assigned largely to the trier of
fact,12
most visibly even if not most frequently the jury, which in engaging in this task is asked
10See Thomas Raeburn White, Oaths in Judicial Proceedings and Their Effect upon the
Competency of Witnesses, 51 U. Pa. L. Rev. 373 (1903); Daniel Blau, Note, Holy Scriptures and
Unholy Strictures: Why the Enforcement of a Religious Orthodoxy Demands a More Refined
Establishment Clause Analysis of Courtroom Oaths, 4 First Amend. L. Rev. 223, 226-29 (2006).
11See 5 John Henry Wigmore, Evidence in Trials at Common Law 32 (James Chadbourn ed.,
1974).
12See, e.g.,United States v. Barnard, 490 F.2d 907, 912 (9
thCir. 1973) (the jury is the lie
detector in the courtroom); State v. Christensen, 163 P.3d 1175 (Idaho, 2007) (holding thatadmission of polygraph evidence would invade the province of the jury); Bloom v. People, 185
P.3d 797, 807 (Colo. 2008) (same); State v. Lyon, 744 P.2d 231, 240 (Or. 1987) (Linde, J.,
concurring) (same). See also United States v. Thompson, 615 F.2d 329, 332-33 (5th
Cir. 1980)
(stating that issues of credibility in general are for jury); State v. Myers, 382 N.W.2d 91, 97
(Iowa, 1986) (same). A comprehensive historical account is George Fisher, The Jurys Rise as Lie
Detector, 107 Yale L.J. 575(1997).
8/22/2019 Can Bad Science Be Good Evidence
9/40
7
to assess, among other things, the demeanor of a witnesses, their past record of truth-telling
(or not), the internal coherence of their stories, and the external coherence of their stories with
the stories of others, all in order to determine who is telling the truth and who is not.
Enter science. Because the criteria traditionally employed by judges and juries to
evaluate the veracity of witnesses have been notoriously unreliable,13
the quest for a scientific
way of distinguishing the truth-teller from the liar has been with us for generations. Indeed,
what for many years was the prevailing legal standard for determining the admissibility of
scientific evidence the Frye test14
arose in 1923 in the context of an unsuccessful attempt to
admit into evidence a rudimentary lie-detection machine invented by William Moulton
Marston. Marston is perhaps better known as the creator of the comic book character Wonder
Woman, whose attributes included possession of a magic lasso, forged from the Magic Girdle of
Aphrodite, which would make anyone it encircled tell the truth without fail. The device at issue
in Frye was a simple polygraph and not a magic lasso, but not only did Frye set the standard for
the admission of scientific evidence for more than half a century, its exclusion of lie-detection
13Useful summaries of the research are in Aldert Vrij, Detecting Lies and Deceit: The Psychology
of Lying and the Implications for Professional Practice (2000); Jeremy A. Blumenthal,A Wipe of
the Hands, A Lick of the Lips: The Validity of Demeanor Evidence in Assessing Witness Credibility,
72 Nebr. L. Rev. 1157 (1993);Lindsley Smith,Juror Ability to Determine Deception and Veracity,
4 Comm. L. Rev. 4 (2000); Olin Guy Wellborn III, Demeanor, 76 Corn. L. Rev. 1075, 1082-88
(1991).
14Frye v. United States, 293 F. 1013, 1014 (D.C. Cir. 1923) (holding that scientific evidence must
use methods generally accepted in the relevant scientific community).
8/22/2019 Can Bad Science Be Good Evidence
10/40
8
technology also paved the way for the continuing exclusion, with few exceptions, of lie-
detection evidence in American courts.15
The science of lie-detection has improved considerably since 1923, but not by so much
as to lead to large-scale changes in judicial attitudes. Indeed, after the replacement (in federal
courts) in 1993 ofFryes general acceptance test by Daubert v. Merrill Dow Pharmaceuticals
Inc.16
and its insistence on various indicia of scientific reliability as a precondition of the
admissibility of evidence purporting to be scientific, the situation has remained much the
same.17
What makes the foregoing important is the rapidly changing state of cognitive
neuroscience the study of human thinking using various methods of (indirectly) measuring
15See, e.g., Brown v. Darcy, 783 F.2d 1389, 1394 (9
thCir. 1986) (excluding lie-detection
evidence); United States v. Gilliard, 133 F.3d 809, 815 (11th
Cir. 1998) (same); United States v.
Sanchez, 118 F.3d 192 (4th
Cir. 1997) (same); Wilkins v. State, 190 P.3d 957 (Kan. 2008) (same);
State v. Jones, 753 N.W.2d 677 (Minn. 2008) (same); People v. Richardson, 183 P.3d 1146 (Cal.
2008) (same). Decisions more sympathetic to the use of polygraph evidence under someconditions include United States v. Cordoba, 104 F.3d 225, 228 (9
thCir. 1997) (holding that a per
se exclusion of polygraph testimony does not survive Daubert); Rupe v. Wood, 93 F.3d 1434 (9th
Cir. 1996) (holding polygraph evidence admissible to support defendants statement at
sentencing hearing); United States v. Posado, 57 F.3d 428 (5th
Cir. 1995) (rejecting per se
exclusion); United States v. Crumby, 895 F. Supp. 1354 (D. Ariz. 1995) (limiting polygraph use to
corroborating or impeaching defendants testimony); United States v. Galbreth, 908 F. Supp.
877, 896 (D.N.M. 1995) (allowing polygraph evidence if examiner is properly qualified);
Commonwealth v. Duguay, 720 N.E.2d 458, 463 (Mass. 1999) (same). The issue is sometimes
dealt with by statute. Such statutes typically exclude polygraph evidence, as in California
Evidence Code 351.1 (2008), but New Mexico is an exception. New Mexico Rules of Evidence
11-707 (2008).
16509 U.S. 579 (1993). Further refinements were added by the Supreme Court in Kumho Tire
Co., Ltd. v. Carmichael, 526 U.S. 137 (1999), and General Electric Co. v. Joiner, 522 U.S. 136
(1997).
17See Christopher B. Mueller & Laird C. Kirkpatrick, Evidence 701-04 (4
thed., 2009).
8/22/2019 Can Bad Science Be Good Evidence
11/40
9
brain activity, as opposed simply to examining the overt behaviors that such brain activity
generates. The tools of modern neuroscience are numerous, but the most prominent of them
is fMRI functional magnetic resonance imaging.18
Commonly called brain scanning, fMRI
examination holds out the possibility of being able to determine which parts of the brain are
being used for which cognitive tasks. Although novices seeing an fMRI scan sometimes believe
that certain parts of the brain light up when engaged in certain tasks,19
what actually occurs is
that the portion of the brain being used recruits more oxygenated blood cells to help it in its
task, and thus what appears to be a lit up part of the brain is actually a part of the brain that has
more oxygenated hemoglobin in it than it had when it was less or differently cognitively
engaged.
As should by now be apparent, the development of fMRI technology has led some
researchers to believe that this technology can be effective in distinguishing liars from truth-
tellers.20
Ifand it is a huge if different parts of the brain are used for lying than for truth
18Accessible explanations of the fMRI include Scott A. Huettel, Allen W. Song, & Gregory
McCarthy, Functional Magnetic Resonance Imaging (2004); Marcus Raichle,A Brief History of
Functional Brain Mapping, in Brain Mapping: The Systems 33 (Arthur W. Toga & John Mazziotta
eds., 2000); Marcus E. Raichle,A Brief History of Human Brain Mapping, 32 Trends in
Neuroscience 118 (2008); Marcus E. Raichle & Mark A. Mintun, Brain Work and Brain Imaging,
29 Ann. Rev. Neuroscience 449 (2006).
19See Teneille Brown & Emily Murphy, Functional Neuroimaging as Evidence of a Defendants
Past Mental States, 62 Stan. L. Rev. (forthcoming 2010), manuscript, at 64.
20See Christos Davatzikos, et al., Classifying Spatial Patterns of Brain Activity with Machine
Learning Methods: Application to Lie Detection, 28 NeuroImage 663 (2005);G. Ganis, et al.,
Neural Correlates of Different Types of Deception: An fMRI Investigation, 13 Cerebral Cortex 830
(2003); Joshua D. Greene & Joseph M. Paxton, 106 Patterns of Neural Activity Associated with
Honest and Dishonest Moral Decisions, Proceedings of the National Academy of Sciences USA
12506 (2009); F. Andrew Kozel, Tamara M. Padgett, & Mark George, Brief Communications: A
8/22/2019 Can Bad Science Be Good Evidence
12/40
10
telling, or for deception rather than honesty, then the possibility exists of using a brain scan to
determine whether the person whose brain is being scanned is lying or telling the truth. Or so
it is claimed. And so especially is it claimed by those who see the commercial potential for just
this technology. For-profit companies, in particular No Lie MRI21
and CEPHOS,22
have already
begun marketing their lie-detection services, and these companies and their principals have
been at the forefront of those touting the courtroom and forensic potential of the new
technology.
Replication Study of the Neural Correlates of Deception, 118 Behavioural Neuroscience 852
(2004); Andrew Kozel, et al., Detecting Deception Using Functional Magnetic Imaging, 58
Biological Psychiatry 605 (2005); Andrew Kozel, et al.,A Pilot Study of Functional Magnetic
Resonance Imaging Brain Correlates of Deception in Healthy Young Men, 16 J. Neuropsychiatry
& Clinical Neuroscience 295 (2004); Daniel D. Langleben, Detection of Deception with fMRI: Are
We There Yet?, 13 Legal Criminological Psych. 1 (2008); Daniel D. Langleben, et al., Telling Truth
from Lie in Individual Subjects with Fast Event-Related fMRI, 26 Human Brain Mapping 262
(2005), pp. 262-72;Daniel D. Langleben, et al., Brain Activity During Simulated Deception: An
Event-Related Functional Magnetic Resonance Study, 15 NeuroImage 727 (2002); Tatia M.C.
Lee, et al., Neural Correlates of Feigned Memory Impairment, 28 NeuroImage 305 (2005);TatiaM.C. Lee, et al., Lie Detection by Functional Magnetic Resonance Imaging, 15 Human Brain
Mapping 157 (2002); Donald H. Marks, Mehdi Adineh, & Sudeepa Gupta, Determination of
Truth From Deception Raising Functional MRI and Cognitive Engrams, 5 Internet J. Radiology
(2006); Feroze Mohamed, et al., Brain Mapping of Deception and Truth Telling abut an
Ecologically Valid Situation: Function MR Imaging and Polygraph Investigation Initial
Experience, 238 Radiology 679 (2006);Jennifer Maria Nunez et al., Intentional False Responding
Shares Neural Substrates with Response Conflict and Cognitive Control, 25 NeuroImage 267
(2005); Sean A. Spence, et al., Speaking of Secrets and Lies: The Contribution of Ventrolateral
Prefrontal Cortex to Vocal Deception, 40 NeuroImage 1411 (2008), pp. 1411-18;Sean A.
Spence, at al., Behavioural and Functional Anatomical Correlates of Deception in Humans, 12
Brain Imaging NeuroReport 2849 (2001);Paul Root Wolpe & Daniel D. Langleben, Lies, Damn
Lies, and Lie Detectors, 86(2) Harv. Bus. Rev. 25 (2008); Paul Root Wolpe, et al., Emerging
Neurotechnologies for Lie_Detection: Promises and Perils, 5 Am. J. Bioethics 39 (2005).
21Seehttp://noliemri.com
22Seewww.cephoscorp.com .
http://noliemri.com/http://noliemri.com/http://noliemri.com/http://www.cephoscorp.com/http://www.cephoscorp.com/http://www.cephoscorp.com/http://www.cephoscorp.com/http://noliemri.com/8/22/2019 Can Bad Science Be Good Evidence
13/40
11
Neuroscience-based lie detection follows a long history of lie-detection technology. The
earliest polygraphs were based on blood pressure, and more modern techniques include
electroencephalography, which measures electric current generated by the brain,23
the analysis
of facial micro-expressions of the kind developed by the psychologist Paul Ekman and now at
the center of the television series Lieto Me,24
periorbital thermography,25
which measures
the temperature around the eyes, and near-infrared spectroscopy,26
which uses infrared light
to measure changes in blood flow and is thus the precursor of the even newer fMRI technology.
These techniques each have their adherents, but I focus here on fMRI primarily because it
appears by all accounts to be the most reliable of these techniques, although plainly not
reliable enough, as we will see, to be endorsed for courtroom or forensic use by the vast
majority of those most familiar with the technology.
Indeed, in legal and policy debates almost as much as in physics, every action appears to
produce an equal and opposite reaction. And so it has been with the reaction of mainstream
academic neuroscientists to the claims regarding the lie-detection potential of fMRI scans. A
prominent article by Stanford law professor Henry Greely and neuroscientist Judy Illes surveyed
23See Lawrence A. Farwell & Sharon S. Smith, Using Brain MERMER to Detect Knowledge
Despite Efforts to Conceal, 46 J. Forensic Sci. 135 (2001).
24Paul Ekman, Telling Lies: Clues to Deceit in the Marketplace, Politics, and Marriage (3rd
ed.,
2001).
25Ioannis Pavlidis & James Levine, Monitoring of Periorbital Blood Flow Rate Through Thermal
Image Analysis and its Application to Polygraph Testing, 2 Engineering Med. & Biol. Socty 1183
(2002).
26See Britton Chance, et al.,A Novel Method for Fast Imaging of Brain Function, Non-Invasively,
with Light,, 1 Optics Express 411 (1998).
8/22/2019 Can Bad Science Be Good Evidence
14/40
12
all of the existing studies of neuroscience-based lie-detection through 2006, concluding that
each of the studies fell far short of existing scientific standards of rigor and that the studies,
both individually and in the aggregate, did not come close to establishing the reliability of fMRI-
based lie-detection.27
Accordingly, Greely and Illes urged a legally imposed moratorium on the
use of the technology for courtroom or forensic purposes until its reliability according to
scientific standards could be established to the satisfaction of a federal regulatory agency.28
Similarly, one leading neuroscientist has insisted that the data offer no compelling evidence
that fMRI will work for lie detection in the real world.29
Another has concluded that *a+t
present we have no good ways of detecting deception despite our very great need for them.30
And still another has concluded that using laboratory findings on fMRI lie-detection in settings
that can potentially impact individuals legal rights should, on the current state of knowledge,
27Henry T. Greely & Judy Illes, Neuroscience-Based Lie Detection: The Urgent Need for
Regulation, 33 Am. J. L. & Med. 377(2007). See also Henry T. Greely, Neuroscience-Based Lie-
Detection: The Need for Regulation,in Using Imaging to Identify Deceit: Scientific and EthicalQuestions 46 (American Academy of Arts and Sciences, 2009); Henry T. Greely, Premarket
Approval Regulation for Lie Detection: An Idea Whose Time May Be Coming, 5 Am. J. Bioethics
50 (2005); Jane Campbell Moriarty, Visions of Deception: Neuroimages and the Search for Truth,
42 Akron L. Rev. 739 (2009); Michael S. Pardo, Neuroscience Evidence, Legal Culture, and
Criminal Procedure, 33 Am. J. Crim. L. 301 (2006).
28Greely & Illes, supra note 27, at 419-20.
29Nancy Kanwisher, The Use of fMRI Lie Detection: What Has Been Shown and What Not, in
Using Imaging to Identify Deceit, supra note 27, at 7, 12. The word compelling is important,
because it sets the standard, or at least Kanwishers standard, for usability. Perhaps there is
plausible evidence, or some evidence, or more persuasive than not evidence, even if
there is not compelling evidence, and the question, to which I shall turn presently, is why
compelling rather than some other standard is the one to be used.
30Marcus E. Raichle,An Introduction to Functional Brain Imaging in the Context of Lie Detection,
in Using Imaging to Identify Deceit, supra note 27, 3, at 6.
8/22/2019 Can Bad Science Be Good Evidence
15/40
13
remain a research topic, instead of a legal tool.31
An editorial in Nature Neuroscience joined
the chorus of skepticism,32
as did a report from a National Research Council committee.33
And
several published articles by researchers and practitioners from various disciplines insisted that
fMRI lie-detection was not ready for the real world.34
Lying at the core of the campaign against the use of fMRI in real world legal settings is
the conviction that the existing state of the research is poor science. And it is poor science, it is
said, not only because of doubts about the reliability rates of neuro-science-based lie detecting
methods, but also because the validity of the research alleged to support those methods and to
determine the announced reliability rates is deeply flawed. The tests that have been
conducted, they critics claim, are different in material ways from real-world lying and truth-
telling, and thus it is a mistake to draw an inference from the accuracy of neural lie-detecting in
experimental settings to the potential accuracy of those methods in detecting real-world liars.35
31
Elizabeth A. Phelps, Lying Outside the Laboratory: The Impact of Imagery and Emotion on theNeural Circuitry of Lie Detection, in Using Imaging to Identify Deceit, supra note 27, 14, at 20.
32Editorial, 11 Nature Neuroscience 1231 (2008).
33National Research Council, Committee on Military and Intelligence Methodology for
Emergent Neurophysiological and Cognitive/Neural Research, Emerging Cognitive Neuroscience
and Related Technologies (2008).
34James R. Merikangas, Functional MRI Lie Detection, 36 J. Am. Acad. Psychiatry & L. 499 (2008);
Michael S. Gazzaniga, The Law and Neuroscience, 60 Neuron 412 (2008); Jed S. Rakoff, Lie
Detection in the Courts: The Vain Search for the Magic Bullet, in Using Imaging to IdentifyDeceit, supra note 27, at 40;Joseph R. Simpson, Functional MRI Lie Detection: Too Good to Be
True?, 36 J. Am. Acad. Psychiatry & L. 491(2008); Sean A. Spence, Playing Devils Advocate: The
Case Against fMRI Lie Detection, 13 Legal & Criminological Psych. 11 (2008).
35Greely & Illes, supra note 27. To the same effect is the argument that *r+eports of finding
brain patterns of activation corresponding to deception almost always use subjects (often
university students) who are told to lie about something (usually a relatively unimportant
8/22/2019 Can Bad Science Be Good Evidence
16/40
14
Part of the difference, indeed the major difference, is that in most instances the experimental
subjects have been instructed to lie, and whether an instructed lie is even a lie at all presents
substantial questions of construct validity whether the experiment measures what it purports
to measure that cast significant doubts on the research conclusions.36
There are also serious
questions about sample size, levels of reliability, potentially confounding variables (whether
subjects are left- or right-handed, for example, or whether the subject moved, factors that
make a difference in evaluating fMRI results), external validity (whether experimental results
would exist in even a parallel real-world situation), and the significant possibility that subjects
could take counter-measures that would render the test results especially unreliable.37
Thus,
the existing research stands accused of being flawed even as pure laboratory research, and far
less applicable to non-laboratory settings than its proponents have typically claimed.
The charges against the existing research go even further. Many of the results have not
been published in peer-reviewed journals and have not been replicated, thus failing to satisfy
the normal standard for assessing scientific outcomes.38
And quite a few of the experiments
indeed, most of them -- have been conducted by those with some connection with No Lie MRI
or CEPHOS and consequently with a commercial interest in the outcome. And, finally, the
matter). Equating the lies told in such an artificial setting to the kind of lies people tell in reality
is pure fantasy at this point. Editorial, supra note 32, at 1231.
36See especially Kanwisher, supra note 29, at 12. See also Greely, Neuroscience-Based Lie-
Detection,, supra note 27, at 50.
37Greely, Neuroscience-Based Lie-Detection,supra note 27, at 50-51; National Research Council,
supra note 33; Raichle, supra note 30, at 6.
38Kanwisher, supra note 29, at 12.
8/22/2019 Can Bad Science Be Good Evidence
17/40
15
alleged degree of accuracy as high as 90%, according to some claims39
of neural lie-
detecting is considerably higher than what the experimental data actually show.40
To the extent that the proponents of neural lie-detecting maintain that their claims
about the reliability and accuracy of their methods are scientifically sound as the product of
scientifically valid experimentation, they appear to have been exposed as relying on flawed
science. Without better evidence of external validity, without dealing with the construct
validity problem of distinguishing the genuine lie from following an instruction to utter words
that are not literally true, without more rigorous scrutiny of the existing claims of reliability,
without higher verified rates of accuracy, without replication, and without subjecting the
research to peer review by financially disinterested scientists, the claimed ability to use fMRI
testing to identify liars appears to be just that a claim and far from what good scientists take
to be a sound scientific conclusion.
III. ON THE RELATIONSHIP BETWEEN SCIENTIFIC AND LEGAL STANDARDSThe fact that the science to date appears both to be methodologically flawed and less
than compelling in its conclusions is far from the end of the story, the arguments of the skeptics
notwithstanding. The rest of the story, however, is not a story about science, but is instead
primarily a story about law,41
and about why there may be good reasons to doubt that the
39Kozel (2005), supra note 20. See also Langleben (2005), supra 20 (76%); Davatzikos, supra
note 20 (89%).
40Greely, Neuroscience-Based Lie Detection,, supra note 27, at 51.
41See Dale A. Nance, Reliability and the Admissibility of Experts, 34 Seton Hall L. Rev. 191, 203
(2003) (arguing for use of legal standards in evaluating scientific expertise).
8/22/2019 Can Bad Science Be Good Evidence
18/40
16
many scientific failings of fMRI-based lie-detection are or should be dispositive for the legal
system.
First, law is about far more than putting criminals in jail, although that particular type of
legal decision appears to generate the fear that motivates so much of the existing scientific
criticism. One of the scientists quoted above has also said that the results would be especially
unreliable if the subject believed that the results could send him to prison,42
and another
participant at the same symposium worried about a future in which the police may request a
warrant to search your brain.43
These may well be legitimate worries, but their seriousness
depends largely on a view of the Fifth Amendment privilege against self-incrimination that
would characterize an involuntary lie-detection test of whatever kind as physical and non-
testimonial,44
an outcome that seems unlikely, albeit not impossible, under current law. Given
that law enforcement authorities may not require a suspect to talk at all, it is difficult to
imagine a legal state of affairs in which a defendants statement is subject to an involuntary
neuroscientific evaluation of its validity, and thus the circumstances in which an involuntary
fMRI would be usable against a defendant would not only require a court to reject an earlier
Supreme Court statement that the results of lie detector tests would be testimonial and thus
42Statement of Nancy Kanwisher, as quoted in A Good Lie Detector is Hard to Find,
http://web.mit.edu/newsoffice/2007/lying.html.
43See Science News, Science Daily, February 19, 2007, at
http://www.sciencedaily.com/releases/2007/02/070218184515.htm.
44Schmerber v. California, 384 U.S. 757 (1964).
http://web.mit.edu/newsoffice/2007/lying.htmlhttp://web.mit.edu/newsoffice/2007/lying.htmlhttp://www.sciencedaily.com/releases/2007/02/070218184515.htmhttp://www.sciencedaily.com/releases/2007/02/070218184515.htmhttp://www.sciencedaily.com/releases/2007/02/070218184515.htmhttp://web.mit.edu/newsoffice/2007/lying.html8/22/2019 Can Bad Science Be Good Evidence
19/40
17
encompassed by the Fifth Amendment,45
but would also require that the fMRI not be used in
conjunction with, or to test the validity of, any statement made by the defendant.46
But even conceding for the sake of argument that a future that included brain scan
warrants is a legitimate worry,47
it is nevertheless a rather large leap to transpose that worry to
a concern about numerous other potential courtroom and forensic uses of lie-detection
technology. Most importantly, the question of what evidence the prosecution can use against a
defendant is very different from the question of what evidence a defendant can use to attempt
to establish his innocence under existing defendant-protective burdens of proof. Suppose, to
attach some arbitrary but conservative numbers to the existing research, that an fMRI
evaluation of a defendants claim of innocence I was somewhere else, or He started the
fight, for example has an accuracy rate of 70%.48
It is of course clear that we should not
imprison people on a 70% probability of their guilt, and we do not do so. But whether to
45Schmerber, 384 U.S. at 764.
46See Benjamin Holley, Its All in Your Head: Neurotechnological Lie Detection and the Fourth
and Fifth Amendments, 28 Developments in Mental Health L. 1 (2009); Sarah E. Stoller & Paul R.
Wolpe, Emerging Neurotechnologies for Lie Detection and the Fifth Amendment, 33 Am. J. L. &
Med. 359 (2007).
47The worry is fueled in part by a case in Mumbai, India, in which an involuntary fMRI scan was
used by the prosecution to challenge the veracity of a criminal defendant. SeeEditorial, supra
note 32.
48Note that the accuracy rate for identifying truth may differ from the accuracy rate foridentifying a lie. Suppose a defendant claims he was somewhere else when the crime was
committed and suppose further that an fMRI indicates he is telling the truth. On the existing
state of the research, see Kanwisher, supra note 29, at 8,this fMRI result is more reliable has
a smaller percentage of errorsthan an fMRI result that indicated that the defendants
statement was false. In other words, truths are identified as lies less often than lies are
identified as truths.
8/22/2019 Can Bad Science Be Good Evidence
20/40
18
imprison people who are 70% likely to be guilty is not the question, or at least not the only
question. An equally important question is whether if there is a 70% chance that a defendants
claim of innocence is accurate, we would want to conclude that his guilt has been proved
beyond a reasonable doubt? Indeed, this was exactly the issue in the 1998 Supreme Court case
ofUnited States v. Scheffer,49
involving a defendant in a military court martial who sought to
introduce in his own defense a polygraph test supporting the accuracy of his assertion of
innocence. The test results had been excluded under Rule 707 of the Military Rules of
Evidence,50
and the defendant challenged the constitutionality under the Due Process51
and
Compulsory Process52
Clauses of Rule 707s absolute exclusion of polygraphic evidence. The
Supreme Court, over Justice Stevens dissent and in the face of at least some concern about a
blanket rule of exclusion expressed by four other Justices who concurred in the judgment but
not with what all of Justice Thomas wrote in announcing the conclusion of the Court, held that
there was no federal constitutional right of a defendant to offer polygraphic exculpatory
evidence. That there may not be such a constitutional right, however, does not answer the
non-constitutional legal policy question of whether such evidence ought to be admitted when
offered by a defendant under these or similar circumstances. Nor does it answer the question
whether the majoritys simple distinction between reliable and unreliable masks the
important distinction between how reliable evidence must be in order to allow its use to a
49523 U.S. 303 (1998).
50Mil. R. Evid. 707.
51U.S. Const. amend. V.
52U.S. Const. amend. VI.
8/22/2019 Can Bad Science Be Good Evidence
21/40
19
defendant who wishes to raise the possibility of a reasonable doubt as to his guilt, whether by
buttressing the defendants claim of innocence or, perhaps even more likely, by attacking the
credibility of a police officer or some other prosecution witness.
Of course any scientific test will have some level of reliability. Whether that level of
reliability is high enough for admissibility, however, depends on the purposes for which it is
being employed. If the outcome of a test is being used as the sole or principal evidence of
whether a defendant should go to prison, as it more-or-less is in some current uses of DNA
identification,53
we should demand extremely high levels of reliability. If the evidence is being
used as merely one component of a larger story about whether a defendant should go to
prison, then perhaps the level of reliability can be lowera brick is not a wall, as the famous
adage in the law of evidence goes.54
After all, it is not true that just because the standard of
proof for conviction of a crime is proof beyond a reasonable doubt that every piece of evidence
admissible to (cumulatively) establish proof beyond a reasonable doubt must be capable of
individually proving beyond a reasonable doubt that the defendant is guilty.55
Nor is it true that
every piece of evidence introduced by the prosecution must be reliable beyond a reasonable
doubt, for such a conclusion would collapse the standard for determining guilt into the
53See, for example, United States v. Beasley, 102 F.3d 1440 (8
thCir. 1996); United States v. Cuff,
37 F. Supp. 2d 279 (S.D.N.Y. 1999); State v. Bartylla, 755 N.W.2d 8 (Minn. 2008). See generally
Peter Donnelly & Richard Friedman, DNA Database Searches and the Legal Consumption of
Scientific Evidence, 97 Mich. L. Rev. 931 (1999).
54McCormick on Evidence 185, at 640-41 (John W. Strong ed., 5
thed., 1999). To the same
effect, it is not to be supposed that every witness can make a home run. Judson F. Falknor,
Extrinsic Policies Affecting Admissibility, 10 Rutgers L. Rev. 574 (1956).
55On confusing admissibility with sufficiency generally, see Dale A. Nance, Conditional
Relevance Reinterpreted, 70 B.U. L. Rev. 447, 449-59 (1990).
8/22/2019 Can Bad Science Be Good Evidence
22/40
20
standard for determining the admissibility of an individual piece of evidence.56
Moreover, if we
are using the evidence to show why a defendant should notgo to prison, then the level of
reliability can be lower still, arguably much lower.57
After all, we do not have a system in which
a defendant goes to prison unless he can prove beyond a reasonable doubt that he is not guilty.
Many of the same considerations apply to civil cases as well. The American legal system
employs the standard of proof by a preponderance of the evidence in almost all civil cases
because the system is committed to the view that failure to award damages, say, to an injured
or otherwise wronged plaintiff is every bit as serious an error as is wrongly requiring a non-
56
See United States v. Glynn, 578 F. Supp. 2d 567, 574-75 (S.D.N.Y. 2008); In re Ephedra
Products Liability Litigation, 393 F. Supp. 2d 181, 187-88 (S.D.N.Y. 2005).
57For an extended argument about the advantages of asymmetry between prosecution and
defense in the standards for admission of scientific evidence, see Christopher Slobogin, Proving
the Unprovable: The Role of Law, Science, and Speculation in Adjudicating Culpability and
Dangerousness 131-44 (2007). The strongest response to the argument for asymmetry is that
the justifiable defendant-skewed epistemic goals of the criminal justice system are already
incorporated in the presumption of innocence and the proof-beyond-a-reasonable-doubt
burden of proof, and to overlay special evidentiary burdens on the prosecution (or special
evidentiary benefits on the defense) would thus be a form of double-counting. See Larry
Laudan, Truth, Error, and the Criminal Law: An Essay in Legal Epistemology 123-28, 144 (2006).
But this argument rests on the assumption, perhaps a justified one but perhaps not, that the
existing standard of proof achieves the socially proper distribution of errors of false acquittal
and of false conviction. If it does not, then, given the historical provenance (and thus resistance
to modification) of the beyond-a-reasonable-doubt-standard, adjusting the results of that
standard (whether upwards or downwards) by the use of other evidentiary, substantive, or
procedural devices seems hardly inappropriate. Nor is there any reason to believe that the best
way to achieve the optimal distribution of error is with one burden of proof rule rather than a
combination of multiple evidentiary and procedural rules. See Raphael M. Goldman & Alvin I.Goldman, Review ofTruth, Error and the Criminal Law: An Essay in Legal Epistemology, by
Larry Laudan, 15 Legal Theory 55 (2009); Michael S. Pardo, On Misshapen Stones and Criminal
Laws Epistemology86 Texas L. Rev. 347 (2007) (book review). Regardless of the outcome of
the debate about asymmetry, however, the very existence of the debate, and the terms on
which it is conducted, demonstrates the folly of trying to determine questions of the legal
usability of evidence without taking account of legal goals and legal standards.
8/22/2019 Can Bad Science Be Good Evidence
23/40
21
culpable defendant to pay damages.58
And once again, therefore, it is hardly clear that a party
in a civil lawsuit seeking to bolster his assertions about facts should be precluded from doing so
by use of a method whose reliability is nowhere near sufficient to send someone to jail.
Awarding damages is less serious than imprisoning someone, or so our legal system believes,
and as long as that is so then it is a mistake to assume that a uniform standard of reliability
should govern all legal uses of a particular type of evidence.
Although the foregoing is about reliability and not validity, the same analysis applies to
questions of experimental validity as well. The experiments alleged to establish the reliability
of fMRI lie-detection have been attacked as lacking in external and construct validity,59
but, like
questions of reliability, these issues of validity are also matters of degree,60
and whether some
degree is or is not good enough will again depend on the uses for which the experiment is being
put. Consider first the question of external validity, the question whether laboratory results on
58On the decision theoretic aspects of burdens of proof in civil cases in general, see James
Brook, Inevitable Errors: The Preponderance of the Evidence Standard in Civil Litigation,18 TulsaL.J. 627 (1982); Bruce Hay & Kathryn Spier, Burdens of Proof in Civil Litigation: An Economic
Perspective, 26 J. Legal Stud. 413 (1997); John Kaplan, Decision Theory and the Factfinding
Process, 20 Stan. L. Rev. 1065 (1968); Dale A. Nance, Civility and the Burden of Proof, 17 Harv.
J.L. & Pub. Pol. 647 (1994); Frederick Schauer & Richard Zeckhauser, On the Degree of
Confidence for Adverse Decisions, 25 J. Legal Stud. 27 (1996). An important challenge to the
conventional view about burdens of proof in civil cases is Ronald J. Allen, The Nature of Juridical
Proof, 13 Cardozo L. Rev. 373 (1991); Ronald J. Allen, A Reconceptualization of Civil Trials, 66
B.U. L. Rev. 401 (1986), arguing, correctly, that because the plaintiff must typically prove each
of multiple elements of an offense by a preponderance of the evidence, then the actual burden
on the plaintiff is substantially higher than the burden on the defendant.
59See notes 27-37 and accompanying text, supra.
60See Kenneth R. Foster & Peter W. Huber, Judging Science: Scientific Knowledge and the
Federal Courts 17 (1999); Erica Beecher-Monas,A Ray of Light for Judges Blinded by Science:
Triers of Science and Intellectual Due Process, 33 Ga. Rev. 1047, 1062 (1999); Nance, supra note
41, at 200.
8/22/2019 Can Bad Science Be Good Evidence
24/40
22
a certain subject population can be employed to reach conclusions or make predictions about a
different and non-laboratory subject population. The issue often arises in psychological
experiments in which conclusions drawn from experiments using university undergraduates a
common pool of experimental subjects can be applied to the behavior of non-undergraduates
in non-laboratory settings. What makes the experimental research useful, however, is that
there are other experiments that have demonstrated a substantial correlation for many kinds of
studies between the results reached in the laboratory with experimental subjects and the
results observed in non-laboratory settings.61
These correlations are not perfect, of course, but
they are positive to a substantial degree, and whether that substantial degree is good enough
will depend on the uses to which the research is to be put. When the research is to be used to
enact policy with negative consequences for some segment of the population, for example, we
should demand a higher correlation between laboratory results and non-laboratory conclusions
than we would if, for example, the population were merely being warned to be aware of a
dangerous phenomenon that thus far had been demonstrated only in laboratory settings on
subjects motivated differently from subjects in the external (non-laboratory) world.
Although perhaps less obvious, the same considerations apply to construct validity as
well. Suppose we wish to determine whether there is a relationship between eating a big
breakfast and increased proficiency in performing mathematical tasks. And then suppose
61See Craig J. Anderson, James J. Lindsay, & Brad J. Bushman, Research in the Psychological
Laboratory: Truth or Triviality, 8 Current Directions in Psychological Sci. 3 (1999); Leonard
Berkowitz & Edward Donnerstein, External Validity is More than Skin Deep: Some Answers to
Criticisms of Laboratory Experimentation, 37 Amer. Psych. 245 (1982); Douglas J. Mook, In the
Defense of External Invalidity, 38 Amer. Psych. 379 (1983).
8/22/2019 Can Bad Science Be Good Evidence
25/40
23
someone were to have conducted an experiment demonstrating a relationship between eating
a big breakfast and an increased ability to avoid misspellings for the rest of that day. If this
experiment were used to support a claim about breakfast and mathematical proficiency, it
would be open to the charge of construct validity, because what the experiment measured
spelling ability was not the same as what the experiment was being offered to show
mathematical proficiency. But if there were to exist a demonstrated correlation between
avoiding spelling mistakes and avoiding mathematical errors, then an experiment showing an
effect on the former would provide some evidence that there would be an effect on the latter.
It would not be conclusive evidence, but the deficiencies in construct validity would not render
the experiment actually conducted totally spurious in terms of the ability to draw conclusions
about a different but correlated effect.
So too, perhaps, with the flaws in construct validity in many existing experiments about
neuroscience-based lie-detection. Although there are important exceptions,62
most of the
experiments purportedly establishing the reliability of fMRI lie-detection are experiments in
which subjects are told by the experimenter to lie or not to lie. When the experimenter is told
to lie, it is argued, following that instruction is not lying at all, and thus an fMRI result
demonstrating a certain kind of brain activity for following an instruction to lie tell us nothing
about the kinds of brain activity involved in actual lying.63
But even though this gap between
the instructed lie and the real lie is a significant problem of construct validity, it would render
62See especially Greene, supra note 20.
63See Greely & Illes, supra note 27; Kanwisher, supra note 29.
8/22/2019 Can Bad Science Be Good Evidence
26/40
24
the experimental results totally useless only if there were no correlation at all between the
brain activity involved in the real lie and that involved in the instructed lie. As yet we do not
know whether such a correlation exists, but if there is some correlation, even if small, once
again it would be incorrect to conclude that the existing studies condemn the use of fMRI-
based lie-detection as completely without evidentiary value, as opposed to offering, say, only
slight evidence.
Although slight evidence ought not to be good enough for scientists,64
it is a large part
of the law. Not only do basic principles of evidence law (as well as human thinking) routinely
allow the cumulation of weak but not spurious pieces of evidence, whether in holistic story-
creation form,65
or for the related purpose of prompting an inference to the best explanation,66
or in Bayesian more linear fashion,67
and not only might weak evidence be sufficient to allow a
defendant to resist a prosecutions claim to have established his guilt beyond a reasonable
doubt, but low standards of proof or persuasion pervade the legal system. A plaintiff in some
states can resist a defendants motion for a directed verdict by offering only a scintilla o f
64See David H. Kaye, Statistical Significance and the Burden of Persuasion, 46 Law & Contemp.
Probs. 13 (1983). And see Raichle, supranote 30, at 5 (equating validity with high statistical
quality).
65See Inside the Juror (Reid Hastie ed., 1993); Reid Hastie, The Role of Stories In Civil Judgments,
32 Univ. Mich. J.L. Ref. 1 (1999);Reid Hastie & Nancy Pennington,A Cognitive Theory of Juror
Decision-Making: The Story Model, 13 Cardozo L. Ref. 519 (1991).
66See Amalia Amaya, Inference to the Best Legal Explanation, in Legal Evidence and Proof:
Statistics, Stories, Logic (H. Kapstein, H. Prakken, & B. Verheij, eds., forthcoming 2009); Ronald
J. Allen & Michael S. Pardo,Juridical Proof and the Best Explanation, 27 L. & Phil. 223 (2008).
67See Alvin Goldman, Quasi-Objective Bayesianism and Legal Evidence, 42 Jurimetrics 108
(2002).
8/22/2019 Can Bad Science Be Good Evidence
27/40
25
evidence.68
In many contexts, evidence that is substantial but less than a preponderance can
generate legal results.69
And the police may stop and frisk a person upon reasonable
suspicion70
and can obtain a search warrant by showing probable cause to believe that the
search will yield usable evidence.71
For all of these purposes, and many more, weak (and thus
potentially flawed) evidence serves pervasive functions in the legal system, and to require that
only compelling or conclusive or even highly reliable evidence, certified as such on the
basis of highly valid scientific processes, would be to dramatically revamp the legal system as
we know it.
IV. JUDGES, JURIES, AND THE DANGERS OF MISUSEIn resisting much of the foregoing argument, it is sometimes said that juries are simply
not very adept at distinguishing one piece of evidence from another, or at evaluating technical
evidence critically, and that as a result superficially persuasive pseudo-scientific evidence will
have more of an effect on deliberations than it should have.72
In effect, the claim is that one
brick will in fact constitute the entire wall for most jurors. Yet even apart from the diminishing
68See, e.g., AALAR, Ltd. v. Francis, 716 So.2d 1141, 1147 (Ala. 1998).
69See, e.g., De la Luente v. FDIC, 332 F.3d 1208, 1220 (9
th
Cir. 2003).70
Terry v. Ohio, 392 U.S. 1 (1968).
71U.S. Const. amend. IV.
72See, for example, Walter Sinnott-Armstrong, Adina Roskies, Teneille Brown, & Emily Murphy,
Brain Images as Legal Evidence, 5 Episteme 359 (2008).
8/22/2019 Can Bad Science Be Good Evidence
28/40
26
role that juries play in American litigation,73
the state of the empirical evidence on jury over-
valuation is decidedly mixed.74
Indeed, were we (and the neuroscientists) to subject the
common claims of jury over-valuation to the same scrutiny that we subject scientific evidence,
we might well find that some of the scientific basis for excluding bad scientific evidence is itself
an example of less than ideal science. A good example is the research purportedly showing
specifically that people will take brain scan images as having more evidentiary value than such
images, in some instances, actually have.75
One of the studies,76
however, compared the effect
of textual neurobabble with that of accurate explanations, another compared brain scans to
73Some of the data are reported in Frederick Schauer, On the Supposed Jury-Dependence of
Evidence Law, 155 U. Penn. L. Rev. 165 (2006).
74See Laudan, supra note 36, at 214-18. Good summaries of the existing primary research,
much of which suggests that juries are not nearly as inept at evaluating scientific or expert
evidence as is often supposed, can be found in, for example, Richard D. Friedman, Minimizing
the Jury Over-Valuation Concern, 2003 Mich. St. L. Rev. 967 (2003); Michael S. Jacobs, Testing
the Assumptions Underlying the Debate About Scientific Evidence: A Closer Look at Juror
Incompetence and Scientific Objectivity,25 Conn. L. Rev. 1083, 1086-93 (1993); Daniel A.Krauss & Bruce D. Sales, The Effect of Clinical and Scientific Expert Testimony on Juror Decision
Making in Capital Sentencing, 7 Psychology, Pub. Pol. & L. 267 (2001);Richard O. Lempert, Civil
Juries and Complex Cases: Taking Stock After Twelve Years,in Verdict: Assessing the Civil Jury
System 181, 235 (Robert E. Litan ed., 1993); Dale A. Nance & Scott B. Morris,Juror
Understanding of DNA Evidence: An Empirical Assessment of Presentation Formats for Trace
Evidence with a Relatively Large and Quantifiable Random-Match Probability, 42 Jurimetrics
403 (2002); Neil Vidmar & Shari S. Diamond,Juries and Expert Evidence, 66 Brook. L. Rev. 1121
(2001). Joseph Sanders, The Paternalistic Justification for Restrictions on the Admissibility of
Expert Testimony, 33 Seton Hall L. Rev. 881, 937-38 (2003), endorses the jury over-valuation
worry, but appears to based it more on problems of complexity of evidence rather than on jury
mis-understanding of science or expertise in non-complex cases.
75D.P. McCabe & A.D. Castel, Seeing is Believing: The Effect of Brain Images on Judgments of
Scientific Reasoning, 107 Cognition 343 (2008); Deena Skolnick Weisberg et al., The Seductive
Allure of Neuroscience Explanations, 20 J. Cognitive Neuroscience 470 (2008).
76Weisberg et al., supra note 75.
8/22/2019 Can Bad Science Be Good Evidence
29/40
27
plain text and to black and white bar graphs,77
and a third compared the effect of a neuroimage
with testimony that was read aloud.78
As a result of failure to exclude the potentially
confounding variables of complexity and of photographic representation and color, however,
we have no idea whether the allegedly distorting effect of brain scans is in fact an effect of
brain scans or is instead the effect of any photographic image,79
or an effect of any image (or
even drawing) in color, or the effect of complex information presented without opposing
explanations and without opportunity for cross-examination. And thus we lack evidence
supporting the belief that judges and juries will overvalue brain scan evidence as such, although
we do have general evidence indicating that jurors may understand more than we think they
do.80
Moreover, in practice if not in theory, the admissibility and use of some types of
evidence, like the actual practice with respect to hearsay and many other exclusions, may vary
with whether it is judge or jury who is serving as the trier of fact. Because juries make none of
the decisions regarding reasonable suspicion to stop, probable cause to search, and other
decisions as to which the credibility of (especially) a police officer is at issue, and because juries
in fact make only a small percentage of the decisions in trials themselves, both criminal and
77McCabe & Castel, supra note 70.
78Jessica R. Gurley & David K. Marcus, The Effects of Neuroimaging and Brain Injury on Insanity
Defenses, 326 Behavioral Sci. & L. 85 (2008).
79See Adina L. Roskies,Are Neuroimages Like Photographs of the Brain, 74 Phil. Sci. 860 (2007);
Sinnott-Armstrong et al., supra note 72, at 367-68. On the distorting effect of photographs
generally, see David A. Bright & Jane Goodman-Delahunty, Gruesome Evidence and Emotion:
Anger, Blame, and Jury Decision Making, 30 L. & Human Behavior 183 (2006). And on the
distorting effect of colored images, see Aura Hanna & Roger Remington, The Representation of
Color and Form in Long Term Memory, 24 Memory & Cognition 322 (1996).
80See authorities cited in note 74, supra.
8/22/2019 Can Bad Science Be Good Evidence
30/40
28
civil, it may be a mistake to extrapolate what we know about jury decision-making or jury
comprehension of scientific evidence to the legal system generally. Admittedly, there would be
difficulties in designing an evidentiary system in which admissibility varied depending on
whether the trier of fact was judge or jury, and formally if not informally the American legal
system has rejected such an approach, but whether what we know about juries, even if the
skepticism about juror comprehension of science is well-grounded, is the appropriate model for
all of law is once again a determination that cannot be made without regard to the normative
goals and varied tasks of the legal system.
That the standards of sufficient reliability should vary with their legal uses remains
subject to the concern about contamination across the different uses, leading ti inappropriate
uses as well as appropriate ones.81
There is a concern, for example, that allowing fMRI lie
detection by a defendant as a way of supporting his claim of innocence might lead to allowance
of the same technique by the prosecution as a way of sending a (possibly innocent) defendant
to prison or by a plaintiff as a way of obtaining damages against a possibly non-culpable
defendant in a civil case. And a related worry would be that allowing judges to hear fMRI lie
detection evidence in evaluating the credibility of a police officer at a suppression hearing
before a judge or magistrate might again lead, eventually, to allowing such evidence to be
heard by juries in determining ultimate guilt or innocence. These worries may not be
completely fanciful, but again the claims are empirical and causal ones about the effect of one
action on another, and it is more than a bit ironic that those who are most insistent about
81See D. Michael Risinger, Navigating Expert Reliability: Are Criminal Standards of Certainty
Being Left on the Dock?, 64 Alb. L. Rev. 99 (2000).
8/22/2019 Can Bad Science Be Good Evidence
31/40
29
finding a sound scientific and empirical basis for the admission of various forms of evidence
seem often to be comfortable abandoning the science in favor of their own hunches when the
question is about the potential downstream dangers of allowing certain forms of evidence to be
used for a particular purpose. Those dangers may well exist, but to date there is no scientific
evidence to support them. For now, the empirical support for the view that allowing fMRI lie
detection by a defendant in a criminal case will lead to allowing fMRI lie detection by the
prosecution against an unwilling defendant appears to be weaker than the admittedly weak
empirical support for the view that fMRI lie detection can actually distinguish liars from truth
tellers.
V. COMPARED TO WHAT?In law as well as in science, one of the most important questions is, compared to
what? And the compared to what? question can usefully be applied to questions about the
determination of witness veracity in courts of law. Traditionally, the task of assessing witness
credibility and veracity has been left to the scientifically-unaided determination of the trier of
fact often the jury but just what are the mechanisms that the trier of fact uses to make
these determinations? We know that jurors often use characteristics other than the content of
what a witness says to evaluate the truth of a witnesss claim, we know that these non-content
characteristics include factors such as whether a witness looks up or down, fidgets, speaks
slowly or quickly, and testifies with apparent confidence, and we know from serious research
that these alleged indicators of veracity are at best highly unreliable, and at worst totally
8/22/2019 Can Bad Science Be Good Evidence
32/40
30
random.82
And this is why numerous studies of the ability of untrained people to determine
truth-telling in others rarely rises above 60%, where 50% would be random.83
Moreover, the
problem of unreliable determination of veracity by judges and juries is exacerbated by the rules
of evidence themselves, which presume on the basis of scarcely more than venerable
superstition that those who have been convicted of serious crimes, even crimes not involving
dishonest statements, are more likely to lie than those who have not,84
which allow witnesses
to offer testimony about whether other witnesses have a reputation in the community for lying
or truth-telling,85
and which allow witnesses to offer their personal opinions about the general
credibility of other witnesses.86
We can thus reframe the question. The question is not whether fMRI based lie
detection is reliable enough in the abstract to be used in court. Rather, it is whether there are
good reasons to prohibit the use of evidence of witness veracity that may well be better than
the evidence of witness veracity that now dominates the litigation process, and at the very least
is probably no worse. The choice is not between very good evidence of veracity and less good
bad, if you will fMRI evidence. Rather it is between admittedly bad fMRI evidence and the
82See authorities cited in note 13, supra.
83See Charles F. Bond, Jr. & Bella M. DePaulo,Accuracy of Deception Judgments, 10 J.
Personality & Soc. Psych. Rev. 214 (2006); Maureen OSullivan, Why Most People Parse Palters,
Fibs, Lies, Whoppers, and Other Deceptions Poorly, in Deception, supra note 8, at 74; Aldert Vrij
et al., Increasing Cognitive Load to Facilitate Lie Detection: The Benefit of Recalling an Event inReverse Order, 32 L. & Human Behavior 253, 253 (2008).
84Fed. R. Evid. 609(a)(1).
85Fed. R. Evid. 608(a).
86Fed. R. Evid. 608(a).
8/22/2019 Can Bad Science Be Good Evidence
33/40
31
even worse evidence that is not only permitted, but in fact forms the core of the common law
trial process.
VI.ON THE USES AND LIMITATIONS OF SCIENCE: DOUBTING DAUBERT
The tone of some of the foregoing notwithstanding, it is decidedly notmy purpose in
this Essay to argue in favor of the admissibility of fMRI-based lie-detection evidence, whether in
the courtroom itself or for related forensic purposes. Rather, it is to suggest that the reliability
and validity standards for scientific evidence that the courts should use must be standards that
come, ultimately, from the legal goals of the legal system and not from the scientific goals of
the scientific system.87
Science can tell us that a certain scientific process has, say, a 12% error
rate, or some rate of false positives and some other rate of false negatives. And scientists must
decide for scientific purposes whether such a rate is sufficient, for example, to assert that
something is the case, to conclude that a finding is adequate for publication, or to find a
research program promising enough for renewal of a research grant. But whether such an
error rate is sufficient for a trier of fact to hear it, put someone in jail, keep someone out of jail,
or support an injunction or an award of damages is not itself a scientific question.
The same applies to the methods of inquiry. Science properly relies on peer-review,
replication, and other indicia of sound methodology. But whether those indicia are the right
ones for purposes of non-scientific action, including but not limited to courtroom verdicts, is
87See Margaret A. Berger, Upsetting the Balance Between Adverse Interests: The Impact of the
Supreme Courts Trilogy on Expert Testimony in Toxic Tort Litigation, 64 L. & Contemp. Prob.
289, 300-02 (2001) (distinguishing legal and scientific standards of causation).
8/22/2019 Can Bad Science Be Good Evidence
34/40
32
not itself a scientific question,88
and to think otherwise is to believe erroneously that one can
derive a legal or policy ought from a scientific is.
Evidence cognoscenti will detect in much of this a challenge to Daubertitself, and that
may be so. Daubertand its successors, General Electric v. Joinerand Kumho Tire Co. v.
Carmichael,89
were directed primarily to the problem of products liability cases and mass tort
verdicts that have been based on allegedly persuasive but seemingly unreliable junk science.90
Indeed, no one who reads the description of the tire failure expert in Kumho Tire can fail to
recognize that junk science really does exist.91
And there is no doubt that the legal system
must guard against a world in which experts in astrology, phrenology, and countless other
bogus -ologies, some of which appear superficially more plausible than astrology and
phrenology but have little more grounding in fact, have a place in the courtroom. Moreover,
and as an important and very recent National Academy of Sciences study has documented in
detail,92
many of the traditionally-used methods of forensic identification methods bitemarks,
88See David L. Faigman et al., Science in the Law: Standards, Statistics and Research Issues 1-
3.5.1, at 43 (2002) (determining value of scientific expert opinion is a matter of policy, not
science).
89See note 16, supra.
90See United States v. Starzepyzel, 880 F. Supp. 1027, 1036 (S.D.N.Y. 1995); Joseph Sanders,
Benedictin on Trial: A Study of Mass Tort Litigation (1998).
91On the junk science problem generally, see Peter W. Huber, Galileos Revenge: Junk Science in
the Courtroom (1991).
92National Research Council of the National Academies, Strengthening Forensic Science in the
United States: A Path Forward (2009).
8/22/2019 Can Bad Science Be Good Evidence
35/40
33
shoe prints, handwriting analysis, ballistics, tool marks, and even fingerprints93
may have less
scientific backing than their proponents have claimed and than the legal system has historically
accepted.94
Identification of the problem is thus comparatively straightforward: prior to the
Daubert revolution and its insistence on reliability and scientific validity,95
American courts
admitted into evidence experts and tests purporting to demonstrate defective manufacture,
causation, or identification, but which were based on empirical findings that do not have a
sound scientific basis as measured by the standards of science. Without Daubert, so the
argument goes, this kind of scientific evidence will continue to be admitted and juries will be
persuaded by it notwithstanding its scientific weakness, causing innocent defendants to be
wrongfully identified and convicted, and non-culpable defendants in tort cases to be held liable.
These steps to what we might call the Daubertconclusion are based on empirical claims
as to which there is little empirical evidence. We do not know, for example, how often the
admission of scientifically substandard evidence has produced an erroneous verdict. It would
do so whenever the admission of substandard evidence was unaccompanied by better evidence
93See United States v. Monteiro, 407 F. Supp. 2d 351, 355 (D. Mass. 2006) (excluding ballistics
testimony); United States v. Green, 405 F.2d 104, 120-22 (D. Mass. 2005) (same). Cf. Robert
Epstein, Fingerprints MeetDaubert: The Myth of Fingerprint Science is Revealed, 75 S. Cal. L.
Rev. 605 (2002) (questioning scientific basis for fingerprint matching).
94See Michael J. Saks, Explaining the Tension Between the Supreme Courts Embrace of Validity
as the Touchstone of Admissibility of Expert Testimony and Lower Courts (Seeming) Rejection of
Same, 5 Episteme 329 (2008); Michael J. Saks, Merlin and Solomon: Lessons from the Laws
Formative Encounters with Forensic Identification Science, 49 Hastings L.J. 1069 (1998).
95Daubert, 509 U.S. at 591 n.9.
8/22/2019 Can Bad Science Be Good Evidence
36/40
34
leading to the same conclusion, whenever the admission of the substandard evidence caused
the trier of fact to find for the prosecution or plaintiff where without that evidence the verdict
would have been otherwise, andwhenever the defendant was not in fact guilty or culpable. It
is possible that there are many cases in which all three of these facts are present, but we do not
know how many there are, or what proportion of all cases, verdicts, accidents, or crimes they
constitute. So it is far from clear how much of a problem there is, and how effective Daubert
has been in solving it. Nor is it clear that the problem of junk science is best solved at the point
of admissibility, because solving it, or at least ameliorating it, by the vigorous use of summary
judgment and dismissal could likely achieve the same result without mistakenly importing the
all-things-considered determination of whether the plaintiff or prosecution should prevail into
the determination of the admissibility of particular pieces of evidence.96
But even ifDauberthas significantly reduced the number of erroneous verdicts actually
caused by poor science, the compared to what question still looms. Bad science is worse
than good science, but not necessarily worse than the non-science that lurks in the heads of
judges and jurors. And flawed science may still be superior to the superstitions and urban
legends that influence so much of public policymaking and legal decision-making. Daubert is
based on the sound premise that manufacturers of products should not be held liable for
damages unless there is a basis for believing that some negligent act of a manufacturer actually
caused injury to the user of the product, but it is important to consider what occurs when bad
science, measured by scientific standards, is excluded from litigation. We do not know with
96See Richard D. Friedman, Squeezing Daubert Out of the Picture, 33 Seton Hall L. Rev. 1047
(2003); Nance, supra note 33, at 252.
8/22/2019 Can Bad Science Be Good Evidence
37/40
35
certainty the answer to this question in all circumstances, but we do know that because
scientific reliability and validity is not a prerequisite for the admission of all evidence, much
non-scientific evidence might well fill the gap left by the excluded flawed scientific evidence.
We do not, after all, have a civil litigation system in which we prohibit those who are injured in
automobile accidents from recovering unless they can show with scientific reliability that the
defendant was driving negligently, nor do we have a criminal litigation system in which we
prohibit defendants from offering a wide variety of non-scientific evidence to keep them out of
prison. Requiring that all legal determinations of guilt or innocence, liability or non-liability, be
guided by science thus seems utopian in both the best and worst senses of that word.97
Best in
the sense that such a system might in fact achieve more justice than the one we now have. But
worst in the sense that eliminating bad or flawed science from the courtroom, the legal system,
and the rules of evidence would require such a dismantling of the entire edifice of common law
adjudication that not only is it fanciful to think it might ever happen, but also that attempting to
make the existing system more scientific by keeping out bad science while not doing anything
about the non-science with which the entire system is infused may produce a system that is in
fact less scientific and less reliable just because it keeps out somewhat poor science in favor of
keeping in the really poor science that sneaks in the back door by not billing itself as science at
all.
97And perhaps that is why Kumho Tire made clear that although Dauberts broad concept of
reliability would be applicable to all expert testimony, using the norms of science to evaluate
reliability would be required only where the proposed evidence or testimony purported to be
scientific. See Glynn and Ephedra, supra note 56.
8/22/2019 Can Bad Science Be Good Evidence
38/40
36
At the heart of the controversy over the legal systems use of poor science may be the
justified concern of scientists to keep theirscientific enterprise as free as possible from non-
scientific taint. When science that is not yet ready for prime scientific time is commandeered
for commercial gain, the enterprise of science suffers, which is exactly what has happened with
the commercialization of fMRI-based lie detection. Encouraging the use of shoddy science for
legal or policy purposes is worse for science, and in the long run may and this too is an
empirical question hurt us all by making science and the use of it by the public and policy-
makers worse. But the tension between the valuable goals of long-term scientific integrity and
short-term uses of scientific output is hardly a new one and hardly unique to lie detection or to
the law of evidence. When medical researchers performing placebo-controlled experiments
reach a point at which they suspect but do not yet know with scientific confidence that a new
drug will cure a fatal disease,98
they face the moral dilemma a dilemma faced by Dr. Martin
Arrowsmith in Sinclair Lewiss great novel99
and by countless real research physicians before
and since -- about whether to sacrifice science to immediate suffering, or instead to sacrifice
peoples health and life to long-term scientific integrity. The stakes with respect to fMRI-based
lie detection may be smaller, but the question is the same. If incomplete or shoddy or
commercially-motivated science is usable in law, science will suffer. But if incomplete or
shoddy or commercially-motivated science is barred from the law in the name of science, laws
98See Robert J. Levine, The Ethics and Regulation of Clinical Research (2
nded., 1986); Sharona
Hoffman, The Use of Placebos in Clinical Trials: Responsible Research or Unethical Practice,?33
Conn. L. Rev. 449 (2001).
99Sinclair Lewis, Arrowsmith (1932).
8/22/2019 Can Bad Science Be Good Evidence
39/40
37
own goals may suffer,100
and the tension and necessary tradeoffs between the goals of law and
the goals of science can never be completely eliminated.101
The claim that evaluating science within the legal system must be based on
characteristically legal standards and norms is in fact an example of a larger issue about the
(partial) distinctiveness of legal thinking, legal analysis, and legal decision-making.102
Judges
routinely based their decisions on precedent and stare decisis, but every elementary textbook
on informal logic treats an argument from past practice as a fallacy. Lawyers are expected to
rely on authority, but thoughtful scientists recognize that reliance on scientific authority is in
great tension with scientific method. When Blackstone observed that it is better that ten
guilty persons escape, than that one innocent suffer,103
he not only drew on ideas now
100Especially important in this context is the obligation of law simply to reach a decision, and
the ability to postpone a judgment until better evidence is available is rarely available to law.
See Neil B. Cohen, The Gatekeeping Role in Civil Litigation and the Abdication of Legal Values in
Favor of Scientific Values, 33 Seton Hall L. Rev. 943 (2003).
101In suggesting that the decision about the use or non-use of neuroscience-based lie-detection
for trial or forensic purposes must be made according to legal standards, I do not mean to
suggest that the decision should be made solely by lawyers and judges. Committees or other
decision-making processes on which both legal and scientific professionals are represented
would be preferable to leaving the decision solely to legal professionals or solely to scientists,
and my principal concern in this paper is only to argue against the view that only scientists