52
23 April, 2012 FOM design & analysis 1 Introduction to study design and analysis for the MFOM dissertation 2012 Reader Department of Public Health University of Oxford Katherine Venables Chief Examiner (Research Methods) FOM

Introduction to study design and analysis for the MFOM ... · 23 April, 2012 FOM design & analysis 14 “Equivalent evidence” •FOM makes the concession that work done for other

  • Upload
    others

  • View
    0

  • Download
    0

Embed Size (px)

Citation preview

Page 1: Introduction to study design and analysis for the MFOM ... · 23 April, 2012 FOM design & analysis 14 “Equivalent evidence” •FOM makes the concession that work done for other

23 April, 2012 FOM design & analysis 1

Introduction to study design

and analysis for the MFOM

dissertation 2012

Reader

Department of Public Health

University of Oxford

Katherine Venables

Chief Examiner

(Research Methods) FOM

Page 2: Introduction to study design and analysis for the MFOM ... · 23 April, 2012 FOM design & analysis 14 “Equivalent evidence” •FOM makes the concession that work done for other

Topics • Welcome

• Why a dissertation for MFOM?

• Process reminder

• “Equivalent evidence”

• Basic questions and timetabling

• Study design

• Interpretation

• Resources

23 April, 2012 FOM design & analysis 2

Page 3: Introduction to study design and analysis for the MFOM ... · 23 April, 2012 FOM design & analysis 14 “Equivalent evidence” •FOM makes the concession that work done for other

Why a dissertation?

23 April, 2012 FOM design & analysis 3

Page 4: Introduction to study design and analysis for the MFOM ... · 23 April, 2012 FOM design & analysis 14 “Equivalent evidence” •FOM makes the concession that work done for other

Why a dissertation?

• 2 population-based specialties, both

require a dissertation

– Occupational Medicine (c 50% by MSc)

– Public Health (100% by MSc)

• Advisory role to employers

– Workplaces vary greatly, some are unique

– Little top-down guidance cf DH, NICE

23 April, 2012 FOM design & analysis 4

Page 5: Introduction to study design and analysis for the MFOM ... · 23 April, 2012 FOM design & analysis 14 “Equivalent evidence” •FOM makes the concession that work done for other

Examples of specialist advice

• Review an employer‟s pre-employment screening

process and make evidence-based recommendations for

revisions

• Investigate an outbreak of unexplained symptoms in one

department of an industrial plant

• Write and implement a pandemic „flu policy for a

workplace, taking account of national guidelines

• Undertake a risk assessment of a new lifting and

handling procedure, draft guidance, audit the outcomes

of the controls, and report to management

• Review the cost-effectiveness of an in-house counselling

service and make recommendations about continuance

23 April, 2012 FOM design & analysis 5

Page 6: Introduction to study design and analysis for the MFOM ... · 23 April, 2012 FOM design & analysis 14 “Equivalent evidence” •FOM makes the concession that work done for other

23 April, 2012 FOM design & analysis 6

Why a dissertation?

• Formulate a clear question

• Find and appraise the evidence

• Present information clearly

• Interpret information, place it in context, make policy recommendations

• Write clearly, succinctly, and logically

• Sustain the motivation to run a longterm project over several months or years from concept to conclusion

• Team-working, networking

Page 7: Introduction to study design and analysis for the MFOM ... · 23 April, 2012 FOM design & analysis 14 “Equivalent evidence” •FOM makes the concession that work done for other

NOT (necessarily) research

• Research skills are a generic GMC

requirement for all specialist training

• MFOM candidates may submit a

substantial audit or systematic review

• No requirement to cover all the curriculum

research competencies in the dissertation

(because these can be covered elsewhere

in the 4 years of training)

23 April, 2012 FOM design & analysis 7

Page 8: Introduction to study design and analysis for the MFOM ... · 23 April, 2012 FOM design & analysis 14 “Equivalent evidence” •FOM makes the concession that work done for other

Process reminder

23 April, 2012 FOM design & analysis 8

Page 9: Introduction to study design and analysis for the MFOM ... · 23 April, 2012 FOM design & analysis 14 “Equivalent evidence” •FOM makes the concession that work done for other

23 April, 2012 FOM design & analysis 9

• Supervisor =educational supervisor, or other(s)

• Protocol and Form M2 (standard dissertations)

• FOM obtains two independent advisory reviews – Rapid feedback

• Final submission and Form M3

• FOM obtains a joint assessment by two assessors – Accept

– Minor revisions

– Major revisions & reassessment

– Reject

• In difficult cases – Additional assessors, vivas

– Appeals process

Page 10: Introduction to study design and analysis for the MFOM ... · 23 April, 2012 FOM design & analysis 14 “Equivalent evidence” •FOM makes the concession that work done for other

23 April, 2012 FOM design & analysis 10

From FOM guidance document

“ … All candidates must accept the responsibility

to produce a dissertation of an acceptable

standard in a timely fashion. …”

Page 11: Introduction to study design and analysis for the MFOM ... · 23 April, 2012 FOM design & analysis 14 “Equivalent evidence” •FOM makes the concession that work done for other

23 April, 2012 FOM design & analysis 11

What are assessors looking for?

• Clarity, including a high standard of grammar, spelling, indexing, referencing & other presentational issues

• < 10,000 words

• Relevance to policy or practice

• Substantial piece of work

• Thoughtful discussion of strengths and weaknesses

Page 12: Introduction to study design and analysis for the MFOM ... · 23 April, 2012 FOM design & analysis 14 “Equivalent evidence” •FOM makes the concession that work done for other

23 April, 2012 FOM design & analysis 12

Assessment proforma • Presentation and layout

• Clarity of abstract

• Background data and references

• Clarity of study aim

• Methodology – appropriateness and clarity

• Statistical methods - appropriateness

• Presentation of results – clarity; tables and figures

• Discussion of findings, potential bias, strengths and weaknesses

• Contextualisation within Occupational Medicine policy or practice

• Statement of contributions

• Ethical issues

• Specific comments (any issues which need addressing or clarifying

Page 13: Introduction to study design and analysis for the MFOM ... · 23 April, 2012 FOM design & analysis 14 “Equivalent evidence” •FOM makes the concession that work done for other

23 April, 2012 FOM design & analysis 13

“Equivalent evidence”

Page 14: Introduction to study design and analysis for the MFOM ... · 23 April, 2012 FOM design & analysis 14 “Equivalent evidence” •FOM makes the concession that work done for other

23 April, 2012 FOM design & analysis 14

“Equivalent evidence”

• FOM makes the concession that work

done for other purpose may be submitted

instead of a MFOM dissertation (and may

be submitted in the same format as for the

other purpose)

– Research dissertation

• MSc Occupational Medicine/Health

• Other MSc/MD/PhD etc

– Substantial published work

• Evidence is assessed by 2 assessors, and

it may need to be revised

Page 15: Introduction to study design and analysis for the MFOM ... · 23 April, 2012 FOM design & analysis 14 “Equivalent evidence” •FOM makes the concession that work done for other

Basic questions and

timetabling

23 April, 2012 FOM design & analysis 15

Page 16: Introduction to study design and analysis for the MFOM ... · 23 April, 2012 FOM design & analysis 14 “Equivalent evidence” •FOM makes the concession that work done for other

Some dissertation project types

• Case series

• Epidemiological studies

• Opinion surveys

• Intervention studies (trials)Qualitative

studies

• Literature reviews

• Audits

• Laboratory-based studies

23 April, 2012 FOM design & analysis 16

Page 17: Introduction to study design and analysis for the MFOM ... · 23 April, 2012 FOM design & analysis 14 “Equivalent evidence” •FOM makes the concession that work done for other

FOM design & analysis 17 23 April, 2012

• Read the journals

• Go to meetings

• Talk to your supervisor

• Talk to your peers

• Read old dissertations

Page 18: Introduction to study design and analysis for the MFOM ... · 23 April, 2012 FOM design & analysis 14 “Equivalent evidence” •FOM makes the concession that work done for other

What is the question?

• Must engage your imagination

• Helps if the answers matter to your

training organisation

• Is your “question” a question, or a design/

method? eg:

– “Study how the results of (a test of a work

competency) change with age”

– Should older workers‟ competency be

assessed more frequently than that of young

workers? Competency to do what task, and

to prevent what adverse outcome? 23 April, 2012 FOM design & analysis 18

Page 19: Introduction to study design and analysis for the MFOM ... · 23 April, 2012 FOM design & analysis 14 “Equivalent evidence” •FOM makes the concession that work done for other

FOM design & analysis 23 April, 2012 19

Page 20: Introduction to study design and analysis for the MFOM ... · 23 April, 2012 FOM design & analysis 14 “Equivalent evidence” •FOM makes the concession that work done for other

Ways of answering questions • Literature review ± meta-analysis

• Observational study

– Survey of current practice and expert opinion

– Epidemiological study

• Longitudinal

• Cross-sectional

• Case-control

– Qualitative study

• Intervention/evaluation study ± economic

evaluation

– Experimental

– Non-experimental

• Clinical audit

23 April, 2012 FOM design & analysis 20

Page 21: Introduction to study design and analysis for the MFOM ... · 23 April, 2012 FOM design & analysis 14 “Equivalent evidence” •FOM makes the concession that work done for other

Should older workers’ competency be assessed

more frequently than that of young workers?

• Literature review ± theoretical simulations

• Survey of SOM members

• Follow-up of a work cohort as it ages

• Survey of variation by age in current workforce

• Comparison of age distribution in cases of competency

“failure” and controls

• Qualitative interviews eg managers, experts, workers

• Intervene (ie assess them more frequently) and evaluate

the outcome

23 April, 2012 FOM design & analysis 21

Page 22: Introduction to study design and analysis for the MFOM ... · 23 April, 2012 FOM design & analysis 14 “Equivalent evidence” •FOM makes the concession that work done for other

23 April, 2012 FOM design & analysis 22

Timetabling • Preparations

– eg reading, discussions, visits, training,

ethics approval, permissions, funding

• Outline protocol FOM

• Data collection and analysis

• Drafting

• Final drafting

• Assessment by FOM

• Revision, resubmission

Page 23: Introduction to study design and analysis for the MFOM ... · 23 April, 2012 FOM design & analysis 14 “Equivalent evidence” •FOM makes the concession that work done for other

It is never too early to think

about document presentation

• Word limit – 10,000 words

• Referencing software

• Indexing

• Pagination

• Appearance of tables and figures

• Photographs

• English style, grammar, spelling

23 April, 2012 FOM design & analysis 23

Page 24: Introduction to study design and analysis for the MFOM ... · 23 April, 2012 FOM design & analysis 14 “Equivalent evidence” •FOM makes the concession that work done for other

Speculate about the likely

study findings

• Implications of range of likely findings

• Skeleton tables and figures

• Headings & sub-headings (IMRAD)

– What did I do?

– How did I do it?

– What does it mean?

23 April, 2012 FOM design & analysis 24

Page 25: Introduction to study design and analysis for the MFOM ... · 23 April, 2012 FOM design & analysis 14 “Equivalent evidence” •FOM makes the concession that work done for other

Study design

23 April, 2012 FOM design & analysis 25

Page 26: Introduction to study design and analysis for the MFOM ... · 23 April, 2012 FOM design & analysis 14 “Equivalent evidence” •FOM makes the concession that work done for other

Basic experimental design

Study

group

Treatment

group

Untreated

group

(controls)

Double-blind follow-up & observations

Ra

nd

om

allo

ca

tio

n

Co

mp

ari

so

n +

te

st

23 April, 2012 FOM design & analysis 26

Page 27: Introduction to study design and analysis for the MFOM ... · 23 April, 2012 FOM design & analysis 14 “Equivalent evidence” •FOM makes the concession that work done for other

Strengths of experimental design

• Random allocation into sub-groups

• Inclusion of untreated control subjects

• Double-blind observation

23 April, 2012 FOM design & analysis 27

Page 28: Introduction to study design and analysis for the MFOM ... · 23 April, 2012 FOM design & analysis 14 “Equivalent evidence” •FOM makes the concession that work done for other

Basic epidemiological design

Target

population

Exposed

group

Comparison

group

Selection & survival

Se

lectio

n &

su

rviv

al

Co

mp

ari

so

n +

te

st

23 April, 2012 FOM design & analysis 28

Page 29: Introduction to study design and analysis for the MFOM ... · 23 April, 2012 FOM design & analysis 14 “Equivalent evidence” •FOM makes the concession that work done for other

Experiment vs observation

• Weaknesses of experiments (eg clinical trials) in human disease

– Specially selected subjects

– Interventions and observations different from “real life”

• Observational studies

– Representative of real people

– Real exposures

– Realistic observations

23 April, 2012 FOM design & analysis 29

Page 30: Introduction to study design and analysis for the MFOM ... · 23 April, 2012 FOM design & analysis 14 “Equivalent evidence” •FOM makes the concession that work done for other

Selection and survival

• Selection into a job

– Workplace factors

– Worker factors

• “Survival” in a job

– Workplace factors

– Worker factors

• Selection into a study

– Availability of records

– Participation

• organisations

• Individuals

– Selection criteria

• inclusion

• exclusion

23 April, 2012 FOM design & analysis 30

Page 31: Introduction to study design and analysis for the MFOM ... · 23 April, 2012 FOM design & analysis 14 “Equivalent evidence” •FOM makes the concession that work done for other

FOM design & analysis

Exposed

group Cases

Non-cases

Unexposed

comparison

group Cases

Non-cases Case Non-

case

Exp+ a b

Exp- c d

Longitudinal, cross-sectional, and case-control designs

23 April, 2012 31

Page 32: Introduction to study design and analysis for the MFOM ... · 23 April, 2012 FOM design & analysis 14 “Equivalent evidence” •FOM makes the concession that work done for other

FOM design & analysis

Measurement:

types of variable

• Determinant

– (independent, stimulus, exposure)

• Outcome

– (dependent, response, disease)

• Modifying variables, including confounders

– eg age, sex, smoking

23 April, 2012 32

Page 33: Introduction to study design and analysis for the MFOM ... · 23 April, 2012 FOM design & analysis 14 “Equivalent evidence” •FOM makes the concession that work done for other

FOM design & analysis

Examples of ways of measuring

occupational exposure • Body burden eg kidney cadmium

• Measured personal exposure eg radiation film badges

• Area measurements eg asbestos fibre counts

• Modelled/estimated exposure

• Job-exposure matrices

• Ordinal scales

• Categories eg job titles

• Duration of job

• Ever/never worked in industry

23 April, 2012 33

Page 34: Introduction to study design and analysis for the MFOM ... · 23 April, 2012 FOM design & analysis 14 “Equivalent evidence” •FOM makes the concession that work done for other

FOM design & analysis

Measurement

• Time relations?

• Natural format/scale of the variable

• Definitions

– Concrete, unambiguous

• Independent data collection

• Information quality

– Valid, repeatable

• Procedures

– Acceptable, safe, practicable

23 April, 2012 34

Page 35: Introduction to study design and analysis for the MFOM ... · 23 April, 2012 FOM design & analysis 14 “Equivalent evidence” •FOM makes the concession that work done for other

FOM design & analysis

Validity and repeatability

23 April, 2012 35

Page 36: Introduction to study design and analysis for the MFOM ... · 23 April, 2012 FOM design & analysis 14 “Equivalent evidence” •FOM makes the concession that work done for other

FOM design & analysis

Validity and repeatability

• Validity: does the index I chose measure

what it is supposed to measure?

– Criterion – compared with the “gold standard”

– Predictive

– Consensus

– Face

• Repeatability: does it give similar findings

on different occasions?

23 April, 2012 36

Page 37: Introduction to study design and analysis for the MFOM ... · 23 April, 2012 FOM design & analysis 14 “Equivalent evidence” •FOM makes the concession that work done for other

FOM design & analysis

Sources of unwanted variation

• Random

• Systematic

• Subject

• Instrument

• Observer } Intra- or inter-

23 April, 2012 37

Page 38: Introduction to study design and analysis for the MFOM ... · 23 April, 2012 FOM design & analysis 14 “Equivalent evidence” •FOM makes the concession that work done for other

FOM design & analysis

Minimising unwanted variation

• Subject

– Design study to minimise sources of variation eg do tests at same time of day

– Multiple tests and use an average

• Instrument

– Same instrument, calibration, adjustment, multiple readings

• Observer

– Eliminate where possible, simple instructions, training, multiple observers

23 April, 2012 38

Page 39: Introduction to study design and analysis for the MFOM ... · 23 April, 2012 FOM design & analysis 14 “Equivalent evidence” •FOM makes the concession that work done for other

Interpretation

23 April, 2012 FOM design & analysis 39

Page 40: Introduction to study design and analysis for the MFOM ... · 23 April, 2012 FOM design & analysis 14 “Equivalent evidence” •FOM makes the concession that work done for other

FOM design & analysis

Overestimation of risk from the

exposure or other determinant

• Information bias

– eg hazardous job → more frequent health

surveillance

– eg symptoms → better recall of exposure

• Confounding

– eg smoking is one of the causal factors for the

disease and is also more common in the

exposed group

23 April, 2012 40

Page 41: Introduction to study design and analysis for the MFOM ... · 23 April, 2012 FOM design & analysis 14 “Equivalent evidence” •FOM makes the concession that work done for other

FOM design & analysis

Underestimation of risk • Small sample size (ie low power)

• Study group of “survivors” in employment

• Chosen population has exposure which is too low, or too short

• Follow-up is shorter than the latent interval

• High random variation in exposure or outcome variable chosen for the study

• Over-estimation of exposure

• Inappropriate comparison group – General population

– Occupational population with other exposures causing the disease

• Confounding 23 April, 2012 41

Page 42: Introduction to study design and analysis for the MFOM ... · 23 April, 2012 FOM design & analysis 14 “Equivalent evidence” •FOM makes the concession that work done for other

Interpretation

• Write for an intelligent, educated layperson

• Respect the reader – grammar, layout etc

• Know your topic well – no excuse for not

knowing about the latest HSE guidance,

the recent Cochrane review, the classic

paper

• Discuss your project‟s strengths and

weaknesses

• Make it clear why your findings matter

23 April, 2012 FOM design & analysis 42

Page 43: Introduction to study design and analysis for the MFOM ... · 23 April, 2012 FOM design & analysis 14 “Equivalent evidence” •FOM makes the concession that work done for other

23 April, 2012 FOM design & analysis 43

Page 44: Introduction to study design and analysis for the MFOM ... · 23 April, 2012 FOM design & analysis 14 “Equivalent evidence” •FOM makes the concession that work done for other

Resources/reserves

23 April, 2012 FOM design & analysis 44

Page 45: Introduction to study design and analysis for the MFOM ... · 23 April, 2012 FOM design & analysis 14 “Equivalent evidence” •FOM makes the concession that work done for other

23 April, 2012 FOM design & analysis 45

Relevant FOM resources

• Searchable database of dissertation abstracts

• Library of accepted dissertations

• (For candidates not undertaking a MSc) feedback on the protocol from 2 reviewers

• Regulations for Membership (MFOM) April 2008

• Question and Answer Sheet on Changes to the Faculty Examinations

• Question and Answer Sheet on Changes to the MFOM Dissertation

• Guidance on Research Dissertations Written for Purpose

Page 46: Introduction to study design and analysis for the MFOM ... · 23 April, 2012 FOM design & analysis 14 “Equivalent evidence” •FOM makes the concession that work done for other

23 April, 2012 FOM design & analysis 46

FOM research competencies:

knowledge Be able to understand:

• How to design a research study.

• How to use appropriate statistical methods.

• The principles of research ethics.

• How to write a scientific paper.

• Sources of research funding.

• The principles and application of epidemiological methods in research and in problem solving

• The application of medical statistics and the interpretation of statistical analysis methods in scientific research.

• Computer based systems for data collection and analysis.

• Ethical considerations in research.

Page 47: Introduction to study design and analysis for the MFOM ... · 23 April, 2012 FOM design & analysis 14 “Equivalent evidence” •FOM makes the concession that work done for other

23 April, 2012 FOM design & analysis 47

FOM research competencies: skills

• Be able to define a problem in terms of needs for an evidence base.

• Be able to undertake systematic literature search.

• Be able to undertake a systematic and critical appraisal and review of scientific literature.

• Be able to produce an evidence based digest of the literature.

• Be able to frame questions to be answered by a research project.

• Be able to develop protocols and methods for research.

• Be able to execute an appropriate study design.

• Plan data collection for simple surveys including sample selection and methods of recording and storing data.

• Be able to use databases.

• Be able to accurately analyse data statistically.

• Have good written and verbal presentation skills.

• Present investigation and results in the format of a research based report.

• Be able to write a scientific paper for peer-reviewed publication.

Page 48: Introduction to study design and analysis for the MFOM ... · 23 April, 2012 FOM design & analysis 14 “Equivalent evidence” •FOM makes the concession that work done for other

23 April, 2012 FOM design & analysis 48

FOM research competencies:

attitudes • Demonstrate curiosity and a critical spirit of enquiry, and

where appropriate a critical attitude towards current practice.

• Acceptance of the need for critical review and for research so as to found a solid base for good practice.

• Ensure patient confidentiality.

• Demonstrate knowledge of the importance of ethical approval and patient consent for clinical research.

• Respect individual confidentiality when presenting data.

• Disposition to cooperation and liaison with statisticians and other research colleagues.

Page 49: Introduction to study design and analysis for the MFOM ... · 23 April, 2012 FOM design & analysis 14 “Equivalent evidence” •FOM makes the concession that work done for other

Bradford Hill criteria for

causation

23 April, 2012 FOM design & analysis 49

Page 50: Introduction to study design and analysis for the MFOM ... · 23 April, 2012 FOM design & analysis 14 “Equivalent evidence” •FOM makes the concession that work done for other

FOM design & analysis

• Strength of association

• Consistency in different studies

• Specificity of exposure, of disease

• Relationship in time

• Biological gradient

• Biological plausibility

• Coherence of all the evidence

• Experimental or semi-experimental

evidence

• Reasoning by analogy 23 April, 2012 50

Page 51: Introduction to study design and analysis for the MFOM ... · 23 April, 2012 FOM design & analysis 14 “Equivalent evidence” •FOM makes the concession that work done for other

FOM design & analysis

• S trength

• C onsistency

• S pecificity

• T ime

• G radient

• P lausibility

• C oherence

• E xperimental

• A nalogy

23 April, 2012 51

Page 52: Introduction to study design and analysis for the MFOM ... · 23 April, 2012 FOM design & analysis 14 “Equivalent evidence” •FOM makes the concession that work done for other

FOM design & analysis

• S trength

• C onsistency

• S pecificity

• T ime

• G radient

• P lausibility

• C oherence

• E xperimental

• A nalogy

• Statistics

• Can

• Sometimes

• Teach

• Good

• Principles,

• Can

• Epidemiology

• Also?

23 April, 2012 52