M.Res Programme

CRITICAL APPRAISAL

Bob Lightowlers, WT Centre for Mitochondrial Research

robert.lightowlers@ncl.ac.uk

NOT EVERYTHING THAT IS PUBLISHED IS CORRECT!!

NOT EVERYTHING THAT IS PUBLISHED IS CORRECT!!

‘The statistical error that keeps on coming’

Ben Goldacre – The Guardian 9th Sept 2011

• Numbers of mice not likely to generate significance• No randomisation• No blinded evaluations • Bias in reporting (no reporting of negative data)• Variable genetic background

• Broader endpoints such as life expectancy/behaviour too crude

Problem transferring ‘effective’drugs from mouse model toclinic

Nonhuman primate models ?

Sept 2011 Bayer Pharmaceuticals assessed reproducibility of 67 published projects

Only 20-25% of published data in line with in-house findings

Jan 2012Amgen tried to confirm research from 53 published ‘landmark’ studies

Jan 2012Amgen tried to confirm research from 53 published ‘landmark’ studies

ONLY 6 (11%) could be confirmed

Jan 2012Amgen tried to confirm research from 53 published ‘landmark’ studies

ONLY 6 (11%) could be confirmed

What causes this lack of reproducibility ?

What causes this lack of reproducibility ?

What causes this lack of reproducibility ?

What causes this lack of reproducibility ?

What causes this lack of reproducibility ?

NOT EVERYTHING THAT IS PUBLISHED IS CORRECT!!

ONLY 15% OF PUBLICATIONS ARE TRUSTWORTHY

• Be critical

• Guilty until proven innocent

The peer review process

What should we be looking for ?

• Incorrect statistical analysis• Power of study• Absence of essential controls• Incorrect methodology• Over/mis interpretation of data• Lack of reference to any conflicting data

What should we be looking for ?

• Incorrect statistical analysis• Power of study• Absence of essential controls• Incorrect methodology• Over/mis interpretation of data• Lack of reference to any conflicting data

• Falsification of data

Somatic Cell Nuclear Transfer

Peer review cannot guarantee scientific integrity

Peer review cannot guarantee scientific integrity

http://www.biochem.arizona.edu/classes/bioc568/papers.htm

Critical Evaluation - a worked example

Human mtDNA

• An autosomally replicating genome

• Found in mitochondrial matrix

• Comprises app. 0.1% of total cell DNA

• Varies enormously in copy number/cell Approx. 700 in fibroblasts to >200,000 in some mammalian oocytes

• Maternally inherited

• Often heteroplasmic in the diseased state

16,569 bp

• Encodes 13 proteins, all of which are OXPHOS components

Hypothesis:

1. Alzheimers Disease could be caused by defects in activity of the respiratory chain complex cytochrome c oxidase

2. Alzheimers Disease is due to mutations in the mitochondrial genome

Why ?

• Lack of FH is a negative risk factor

Why ?

• Lack of FH is a negative risk factor • Risk of AD increases with affected maternal relative (mtDNA?)

Why ?

• Lack of FH is a negative risk factor • Risk of AD increases with affected maternal relative (mtDNA?)

• Mutations in mtDNA can lead to defective OXPHOS

Why ?

• Lack of FH is a negative risk factor • Risk of AD increases with affected maternal relative (mtDNA?)

• Mutations in mtDNA can lead to defective OXPHOS

• Neurons may be particularly susceptible to such defects

DiabetesThyroid Disease

Myopathy

Peripheral Neuropathy

Deafness

CVA / Seizures /

Developmental delay

Respiratory Failure Optic Atrophy / Retinitis Pigmentosa

Cardiomyopathy

Short StatureMarrow Failure

Liver Failure

NeurologicalNon-Neurological

Why ?

• Lack of FH is a negative risk factor• • Risk of AD increases with affected maternal relative (mtDNA?)

• Mutations in mtDNA can lead to defective OXPHOS

• Neurons may be particularly susceptible to such defects

• COX activity reported to decrease in brain of AD patients

Methods used

• MtDNA isolation and sequencing in patients, asymptomatic relatives and controls

Methods used

• MtDNA isolation and sequencing in patients, asymptomatic relatives and controls

• All three COX genes sequenced

Methods used

• MtDNA isolation and sequencing in patients, asymptomatic relatives and controls

• All three COX genes sequenced

• Platelet fusion from AD patients to neuronal cells lacking mtDNA (rho0)

Generation of transmitochondrialcybrids

Biopsy

EthBr Enucleation

Methods used

• MtDNA isolation and sequencing in patients, asymptomatic relatives and controls

• All three COX genes sequenced

• Platelet fusion from AD patients to neuronal cells lacking mtDNA (rho0)

• Analysis of respiratory enzyme activity in the cybrids

Methods used

• MtDNA isolation and sequencing in patients, asymptomatic relatives and controls

• All three COX genes sequenced

• Platelet fusion from AD patients to neuronal cells lacking mtDNA (rho0)

• Analysis of respiratory enzyme activity in the cybrids

• Analysis of ROS production in cybrids

Results

506 Patients and 95 controls

Results

506 Patients and 95 controls

10 clones of all three COX genes sequence

Results

506 Patients and 95 controls

10 clones of all three COX genes sequence

6 mutations found in COI and COII

Results

506 Patients and 95 controls

10 clones of all three COX genes sequence

6 mutations found in COI and COII

Different levels of heteroplasmy but levels significantly greater in the AD cohort

Results

506 Patients and 95 controls

10 clones of all three COX genes sequence

6 mutations found in COI and COII

Different levels of heteroplasmy but levels significantly greater in the AD cohort

No disease-associated mutations in COIII gene

Results

506 Patients and 95 controls

10 clones of all three COX genes sequence

6 mutations found in COI and COII

Different levels of heteroplasmy but levels significantly greater in the AD cohort

No disease-associated mutations in COIII gene

AD cybrids but not controls had low COX activity

Results

506 Patients and 95 controls

10 clones of all three COX genes sequence

6 mutations found in COI and COII

Different levels of heteroplasmy but levels significantly greater in the AD cohort

No disease-associated mutations in COIII gene

AD cybrids but not controls had low COX activity

Increased production of ROS in AD cybrids

Critical evaluation:

How appropriate and robust are the methods ?

Is the data and evaluation robust ?

Are the conclusions valid, based on the reported data ?

Has the paper been validated by other labs and do the authors self-cite excessively ?

How does the paper stand the test of time ?

Is there any conflict of interest ?

Why has this not been reported before ?

mtDNA is destroyed by boiling

We’re looking at pseudogenes!

How can an apparent increase in nuclear pseudogene mutationspredispose to Alzheimers Disease and cause respiratory chaindeficiency in transmitochondrial cybrids ?

Critical evaluation:

How appropriate and robust are the methods ?

Is the data and evaluation robust ?

Are the conclusions valid, based on the reported data ?

Has the paper been validated by other labs and do the authors self-cite excessively ?

How does the paper stand the test of time ?

Is there any conflict of interest ?

How can an apparent increase in nuclear pseudogene mutationspredispose to Alzheimers Disease and cause respiratory chaindeficiency in transmitochondrial cybrids ?