Quantitative Research

Ziba Nadimi- Outreach Library Service Manager CNWL-NHS Camden Provider Services

June 2012

Learning Objectives

explain what is meant by the term critical appraisal

and why it is important

discuss the major features of quantitative research

and the key terminology associated with it

use a CASP checklist to critically appraise a

quantitative research article

What is critical appraisal?

“It usually comes as a surprise to students to learn

that some (the purists would say up to 99% of)

published articles belong in the bin and should not be

used to inform practice” (Greenhalgh 2006)

Not all papers are equal: some are good, some are

bad, most have strengths and weaknesses

“critical appraisal is concerned with the acquisition of necessary skills with which to discern clinical research papers accurately” (Ajetunmobi, 2002)

Understanding (in-depth reading)

CASP International-http://www.caspinternational.org/?o=1012

Dawes (2005) on critical appraisal

It may be easier to try to do it with someone else

It is hard to appraise a paper if you have not read it

Start with easy questions

Carry on even if you find flaws (there may still be useful information in the paper)

Look out for bias!

Review your appraisal – is it balanced, or have you “rubbished” the paper?

Can the results of the research be applied locally?

Quantitative vs. Qualitative Research?

“Both methods are valid if applied to appropriate

research questions, and they should complement

each other” (Bowling, 2002)

“The view that the two approaches are mutually

exclusive has itself become „unscientific‟”

(Greenhalgh, 2006)

Statistics

“…critical appraisal should not…be regarded as just another aspect of medical statistics. This misconception often leads to an overestimation of the level of statistical knowledge required for critical appraisal.” (Ajetunmobi, 2002)

However, for quantitative papers a little statistical knowledge can help a lot!

Definition

“…quantitative research should begin with an idea

(usually articulated as a hypothesis) which then,

through measurement, generates data and, by

deduction, allows a conclusion to be drawn.”

(Greenhalgh, 2006)

Quantitative Data Collection

Requires a specific protocol

Protocol is specified in advance of data collection.

Sample should be large. The larger the better.

Data Analysis

Statistical analysis

It should describe trends, compare groups &

relate variables

The analysis should also compare results with

past research

The Anatomy of a Research Paper

Introduction

Methods

Results and

Discussion

The Anatomy of a Research Paper

The introduction summarises the background to the

study - why

The methods describe how data was gathered and

analysed – vital for the critical appraisal of a paper

The results report findings objectively without

speculation or interpretation - what

In the discussion the authors interpret the findings

in light of the study design and other research. They

may also discuss limitations of their work – what it

means

Levels of Evidence – a general guide

Systematic Review

Identifies a health care intervention and examines the evidence as to whether or not this intervention works.

The authors locate, appraise and synthesise evidence from as many relevant scientific studies as possible.

They summarise conclusions about effectiveness.

They provide a collation of the known evidence on a given topic and identify gaps in research.

Statistical methods (meta-analysis) may or may not be used to analyse and summarise the results.

The Cochrane Library www.thecochranelibrary.com

Advantages of Systematic Reviews

Adhere to a strict design, therefore minimise the

chance of bias.

Provide a scientific rather than subjective

summarisation of literature.

Allow large amounts of information to be assimilated

quickly by healthcare providers, researchers, and

policymakers

Compare results of different studies to establish

generalisability of findings and consistency of results.

Disadvantages of Systematic Reviews

SRs are not the best way to address every question a

Cochrane reviews only include clinical trials

Papers with more interesting results are more likely

to be published (publication bias).

SRs may be biased due to the exclusion of relevant

studies (poor literature search) and the inclusion of

inadequate studies (no quality assessment).

SRs include an element of judgement

Randomised Controlled Trials- RCTs

Two or more interventions are compared by being

randomly allocated to participants.

Includes a control intervention or no intervention.

If possible should be single/double blinded.

Blinding in RCTs

Preventing those involved in a trial from knowing to

which comparison group, i.e. experimental or control,

a particular participant belongs.

The risk of bias is minimised.

Participants, caregivers, outcome assessors and

analysts can all be blinded.

Blinding of certain groups is not always possible. e.g.

surgeons in surgical trials.

Cohort Studies

An observational study

A defined group of people (the cohort) is followed

over time.

Outcomes are compared to examine people who

were exposed or not exposed to a particular

intervention.

A retrospective cohort study - identifies subjects from

past records and follows them to the present.

A prospective cohort study - assembles participants

and follows them into the future.

Case Control Studies

Compares people with a specific disease or outcome

of interest (cases) to people from the same

population without that disease or outcome (control).

Seeks associations between the outcome and prior

exposure to particular risk factors e.g. one group may

have been exposed to a particular substance that the

other was not.

They are usually concerned with causes of a disease.

They are usually retrospective.

Case Series

A study reporting observations on a series of

individuals, usually all receiving the same

intervention, with no control group.

Statistics

A small knowledge of statistical concepts will be

sufficient to appraise most quantitative papers

These include:

Risk (expressed as odds ratios or weighted mean

difference)

Confidence intervals

P values

This is a diagram from a Cochrane Review. It shows the risk ratios

(odds ratios) for a number of different clinical trails.

The vertical line has a ratio of one. This represents a point where

the treatment and comparison are the same. It is known as the

line of no effect.

Results to the left of the line of no effect = less of the outcome in the

experimental group. Odds ratio <1.

Results to the right of the line of no effect = more of the outcome in the

experimental group. Odds ratio >1.

It is important to note whether the outcome is good or bad.

The horizontal line is the confidence interval for each study. It usually represents 95% of the population sampled.

If CI crosses line of no effect = Inconclusive results.

Longer CI = Smaller study (less confident of results).

Shorter CI = Bigger study (more confident of results).

In a systematic review the diamond shows the meta-analysis: it

represents the statistical summary of the separate studies. This is

only possible if the studies are sufficiently similar (homogenous).

Weighted Mean Difference

The design of the diagram is very similar, but the “line of no effect” is set at zero. Results to the left of the line are expressed in negative numbers, for example -0.5

P Values

P for probability (ranging from zero to one).

The result could have occurred by chance if in reality

the null hypothesis was true.

The null hypothesis- the factor of interest (e.g.

treatment) has no impact on outcome (e.g. risk of

death).

P value of less than 0.05 means the likelihood of

results being due to chance is less than 1 in 20 =

“Statistically significant”.

References

•McGovern, D.P.B. etal, Evidence-based medicine, BIOS Scientific Publishers Ltd., 2001

•Greenhalgh, T., How to read a paper, 3rd ed., BMJ Publishing Group, 2006

•Kelsey, K.D., (Lecture 2) Quantitative and qualitative approaches to research - PowerPoint

Presentation

•Ward, L., Critical reading made easy:effectiveness and experience, University Hospitals of

Leicester NHS Trust- PowerPoint Presentation

•Jackson, N., Conducting systematic reviews of public health and health promotion interventions,

Cochrane Health Promotion and Public Health Field- PowerPoint Presentation

•The Cochrane Library‟s Glossary of Terms, Wiley InterScience

•Andrew Hayward, Critical Appraisal of Analytical Studies - PowerPoint Presentation

•Ajetunmobi, O. (2001) Making sense of critical appraisal London: Arnold

•Dawes, M. (2005) „Introduction to critical appraisal‟. in Dawes, M et al Evidence-based practice

(2nd ed) Edinburgh: Elsevier

•Sackett et al (1996) „Evidence based medicine: what it is and what it isn't‟ BMJ 312: 71-72

(13 January)

•Freshwater, D. (2005) Blackwell‟s Nursing Dictionary (2nd ed) Oxford: Blackwell

•Bowling, A. (1997) Research methods in health Buckingham: Open University Press