The Scientific Literature:The Scientific Literature:Scientific AppraisalScientific Appraisal

John BycroftJohn Bycroft

Sarah KnightSarah Knight

Purpose of Scientific LiteraturePurpose of Scientific Literature

• Communication between researchersCommunication between researchers

• Historical record of progress of scientific Historical record of progress of scientific researchresearch

• Avoids duplication of results and methodsAvoids duplication of results and methods

• No Journal of Negative Results so some No Journal of Negative Results so some unsuccessful studies may be repeated.unsuccessful studies may be repeated.

Why Read the Scientific Literature?Why Read the Scientific Literature?

• Keeping up with current scientific / clinical findingsKeeping up with current scientific / clinical findings• Evidence-based medicine decisionsEvidence-based medicine decisions• Developing new methodologiesDeveloping new methodologies• Find deficiencies in current knowledgeFind deficiencies in current knowledge

– Preparation of a literature review for a thesisPreparation of a literature review for a thesis– Writing a review articleWriting a review article– Reviewing a paper for a journalReviewing a paper for a journal

Why is Critical Appraisal Important?Why is Critical Appraisal Important?

• Determine the true results and conclusionsDetermine the true results and conclusions

• Assess the validity of the resultsAssess the validity of the results

• Assess the relative importance of the studyAssess the relative importance of the study

Types of PublicationTypes of Publication

• Peer reviewed journalsPeer reviewed journals– Specialist (Urology, BJUI)Specialist (Urology, BJUI)

– Populist (Nature, Science etc)Populist (Nature, Science etc)

• In-house publicationsIn-house publications

• Conference proceedingsConference proceedings– Peer reviewedPeer reviewed

– Non-peer-reviewedNon-peer-reviewed

• BooksBooks

• ThesesTheses

Types of ArticleTypes of Article

• Original ArticleOriginal Article

• Review ArticleReview Article

• Technical NoteTechnical Note

• Letters to Editor Letters to Editor

• Short CommunicationShort Communication

• Case ReportCase Report

The Scientific MethodThe Scientific Method

• Anecdote or ObservationAnecdote or Observation

• Literature ReviewLiterature Review

• HypothesisHypothesis

• Design of experiment under controlled conditions to Design of experiment under controlled conditions to test hypothesistest hypothesis

• Disproof or supportDisproof or support

• PublicationPublication

• Hypothesis Hypothesis Theorem Theorem

Anatomy of a PaperAnatomy of a Paper

• AbstractAbstract

• Introduction/BackgroundIntroduction/Background

• Methods & MaterialsMethods & Materials– Patient SelectionPatient Selection

• ResultsResults

• DiscussionDiscussion

• ConclusionConclusion

AbstractAbstract

• Summary of problem and main resultsSummary of problem and main results

• Beware of assuming results from reading the Beware of assuming results from reading the abstract alone. This may not have been written abstract alone. This may not have been written by the authors and may be misleading or not by the authors and may be misleading or not contain all useful findings of the studycontain all useful findings of the study

• You can’t appraise a paper from the abstract You can’t appraise a paper from the abstract alonealone

IntroductionIntroduction

• Understanding and presentation of previous Understanding and presentation of previous work (by others and themselves) work (by others and themselves)

• Clear statement of problem and hypothesisClear statement of problem and hypothesis

• How they intend to answer problem or prove How they intend to answer problem or prove hypothesishypothesis

Methods & MaterialsMethods & Materials

• Detailed enough to allow repetition of studyDetailed enough to allow repetition of study

• Experimental DesignExperimental Design

• Statistical TestsStatistical Tests– A prioriA priori– P value for null hypothesisP value for null hypothesis

• Repeatability and reproducibilityRepeatability and reproducibility

Experimental DesignExperimental Design• Test and ControlsTest and Controls

• Sample sizeSample size– power equationpower equation

• Patient SelectionPatient Selection– Inclusion/exclusion criteriaInclusion/exclusion criteria

– RandomisationRandomisation

– Matching (age/sex)Matching (age/sex)

• Animal modelAnimal model

• Study DesignStudy Design– Longitudinal/RetrospectiveLongitudinal/Retrospective

– Placebo controlled etcPlacebo controlled etc

Validation of MethodsValidation of Methods

• Specificity, precision, reliabilitySpecificity, precision, reliability

• Error of measurementsError of measurements

• Subjective or objectiveSubjective or objective

• inter-, intra-rater variabilityinter-, intra-rater variability

• BiasBias

• Diurnal or hormonal variationsDiurnal or hormonal variations

ResultsResults

• Appropriate presentationAppropriate presentation– GraphicalGraphical– TabularTabular

• Statistical AnalysesStatistical Analyses

• Factual statements about observationsFactual statements about observations

DiscussionDiscussion

• Brief summary of methodsBrief summary of methods

• Explanation of limitationsExplanation of limitations

• Implications of the results, individually and Implications of the results, individually and collectivelycollectively

• Reference to previous studiesReference to previous studies

• Overall conclusionOverall conclusion

ConclusionsConclusions

• Supported by experimental resultsSupported by experimental results

• ObjectiveObjective

• Answer the hypothesis originally statedAnswer the hypothesis originally stated

Key Points for Scientific AppraisalKey Points for Scientific Appraisal

• What are you trying to achieve?What are you trying to achieve?• Read through completelyRead through completely• Find statement of problemFind statement of problem• Find key results and conclusions Find key results and conclusions • Determine strengths and weaknesses, e.g. major Determine strengths and weaknesses, e.g. major

experimental design flawsexperimental design flaws• Are conclusions supported by results?Are conclusions supported by results?• How are the conclusions or other findings relevant to youHow are the conclusions or other findings relevant to you

Typical Journal Reviewers ChecklistTypical Journal Reviewers Checklist

• Comments on the novelty of these findingsComments on the novelty of these findings

• Do the findings of this study represent a major Do the findings of this study represent a major (rather than incremental) advance over (rather than incremental) advance over previously published observations?previously published observations?

• Is this study hypothesis driven?Is this study hypothesis driven?

Courtesy of the Journal of Applied Physiology

• Is the rationale for this study logically Is the rationale for this study logically presented?presented?

• Are there major concerns regarding the Are there major concerns regarding the experimental design and methods?experimental design and methods?

• Are the results clearly presented?Are the results clearly presented?

• Can the discussion be improved?Can the discussion be improved?

Typical Journal Reviewers ChecklistTypical Journal Reviewers Checklist