What counts as good evidence?

Dylan EvansDepartment of Mechanical Engineering

University of Bath

It’s hard to know what is bogus!

• NASA invests $1m in antigravity machine

• US Patent 6,362,718 issued for a motionless electromagnetic generator

• Both of these machines are physically impossible

• But scientists have advocated both!

Seven warning signs

• Robert L Park is a Professor of Physics at the University of Maryland

• He has identified seven warning signs that indicate that a scientific claim is bogus.

• Remember, these are only warning signs – they are not infallible.

1. Pitching to the media

• The integrity of science rests on the willingness of scientists to expose new ideas and findings to the scrutiny of other scientists before anyone else.

• This is called ‘peer review’.• Going to the media first suggests that

the work is unlikely to stand up to peer review. Eg. Pons & Fleischmann, 1989.

2. Conspiracy and cover-ups

• The discoverer says that a powerful establishment is trying to suppress his or her work.

• Mainstream science is made out to be a part of a larger conspiracy that includes industry and the government.

• Examples: GM potatoes, cold fusion.

3. At the limit of detection

• The scientific effect is said to be very hard to detect.

• All scientific measurements must contend with some level of background noise.

• But if the signal-to-noise ratio cannot be improved, even in principle, the effect is probably not real.

• Examples: parapsychology.

4. Evidence is anecdotal

• Anecdotes and narrative have a very strong emotional impact.

• Statistics have a weak emotional impact.

• Frauds tend to use anecdotes and not statistics such as RCTs.

• Scientists have to learn to put statistics above anecdote. Eg. bloodletting.

5. Ancient wisdom

• The discoverer says a belief is credible because it has endured for centuries.

• ‘Our ancestors possessed wonderful advanced knowledge that has been forgotten.’ Eg. crystal healing.

• Science was born when people began to reject blind tradition in favour of experiment and observation (eg. Bacon)

6. Splendid isolation

• The image of the lone genius who struggles in secrecy in an attic laboratory is rarely accurate.

• Scientific breakthroughs nowadays are almost always syntheses of the work of many scientists.

7. New laws of nature

• The discoverer proposes a new law of nature that conflicts with what is already well-established.

• Weigh up the evidence for each alternative: eg. homeopathy and the meta-analysis in the BMJ.

On the other hand…

• Sometimes a scientist MAY go to the media first!

• Sometimes there ARE powerful forces trying to suppress a good idea!

• Some scientific effects ARE very hard to detect!

On the other hand… (contd)

• Anecdotal evidence is better than NO evidence at all!

• Some ancient knowledge HAS been forgotten!

• Some scientists DO work in isolation!

• Sometimes a new law of nature IS correct!

So how can we tell?

• No ‘recipe’ or ‘algorithm’ for assessing science.

• No substitute for thinking!

• But there are some general guidelines on how to be critical. Apply them to your own work and to that of others.

Whenever you read a paper…..(even your own!)..

• What is the hypothesis being tested?

• Read the methods section – is there enough information to allow the experiment or observation to be repeated?

• Read the results section – is there a proper statistical analysis?