Research CurriculumResearch CurriculumSession II –Study Subjects, Session II –Study Subjects,
Variables and Outcome MeasuresVariables and Outcome Measures
Jim Quinn MD MSJim Quinn MD MS
Research Director , Division of Emergency MedicineResearch Director , Division of Emergency Medicine
Stanford UniversityStanford University
OverviewOverview
Study SubjectsStudy Subjects- SamplingSampling
- RecruitmentRecruitment
VariablesVariables- Types of outcome measuresTypes of outcome measures
- Precision, accuracy, validity, reliabilityPrecision, accuracy, validity, reliability
Study SubjectsStudy SubjectsGeneralizing the ResultsGeneralizing the Results
““Research is only interesting to others if Research is only interesting to others if they can apply it to their practice”they can apply it to their practice”
Study SubjectsStudy Subjects
Subjects in the study sample should be Subjects in the study sample should be representative of the population of interestrepresentative of the population of interest
Depending on study different populations Depending on study different populations may yield different results.may yield different results.
- Examples: General population, ED Examples: General population, ED patients, Clinic Patients, Attitudes of patients, Clinic Patients, Attitudes of patientspatients
- Laceration studies, syncope studyLaceration studies, syncope study
Study SubjectsStudy Subjects
Specify the best clinical and demographic Specify the best clinical and demographic characteristics of the study population to characteristics of the study population to best answer questionbest answer question
Appropriate sampling from that target Appropriate sampling from that target populationpopulation
Results = truth in the study Results = truth in the study
Best possible chance to have the results Best possible chance to have the results generalizable.generalizable.
Selection CriteriaSelection CriteriaDefining the Target PopulationDefining the Target Population
Inclusion CriteriaInclusion Criteria
- defines the main characteristics of the defines the main characteristics of the target population – be specifictarget population – be specific
Selection CriteriaSelection Criteria Defining the Target PopulationDefining the Target Population
Exclusion CriteriaExclusion Criteria
- Individuals whose characteristics may Individuals whose characteristics may interfere with the quality of the resultsinterfere with the quality of the results
E.g. – rare events, poor follow-upE.g. – rare events, poor follow-up
- May compromise generalizability of the - May compromise generalizability of the studystudy
SamplingSampling
Convenience SampleConvenience Sample
Consecutive SampleConsecutive Sample
Probability SamplesProbability Samples
- Simple Random Sample- Simple Random Sample
- Stratified Random SampleStratified Random Sample
- Cluster SamplesCluster Samples
RecruitmentRecruitmentGoalsGoals
A sample that represents the target A sample that represents the target populationpopulation
- Non responders, lost follow-up- Non responders, lost follow-up
Enough subjects to meet sample size Enough subjects to meet sample size requirementsrequirements
- Play it safe, overestimate- Play it safe, overestimate
- There is always fewer patients than you - There is always fewer patients than you think!think!
““The best way to eliminate disease The best way to eliminate disease is to study it!”is to study it!”
Outcome MeasuresOutcome Measures
Selection of Variables and ScalesSelection of Variables and Scales
Selection of VariablesSelection of VariablesPractical Points/Precision/AccuracyPractical Points/Precision/Accuracy
Continuous VariablesContinuous Variables- ““discrete” variablesdiscrete” variables- rich in informationrich in information- Potential sample size “relief”Potential sample size “relief”
CategoricalCategorical- DichotomousDichotomous- NominalNominal- Ordinal Ordinal
Measurement ScalesMeasurement Scales
Categorical VariablesCategorical Variables
- Phenomena often not suited for Phenomena often not suited for measurement (e.g. Death)measurement (e.g. Death)
- DichotomousDichotomous
- NominalNominal
- Ordinal – categories have order but no Ordinal – categories have order but no specific numerical or uniform difference specific numerical or uniform difference
Measurement ScalesMeasurement Scales
Continuous (infinite values)Continuous (infinite values)
Ordered discrete (ordinal with numerical Ordered discrete (ordinal with numerical meaning)meaning)
- Statistically handled very similarly- Statistically handled very similarly
Measurement ScalesMeasurement ScalesSummarySummary
CategoricalCategorical
- Scales may have more meaning and make Scales may have more meaning and make more sense. more sense.
- Less information, need large numbersLess information, need large numbers
ContinuousContinuous
- some times hard to determine meaningful some times hard to determine meaningful differencesdifferences
- sample size friendlysample size friendly
Attributes of Outcome MeasuresAttributes of Outcome MeasuresPrecisionPrecision
Is the measure “reproducible, reliable and Is the measure “reproducible, reliable and consistent”consistent”
Subject to random error and variabilitySubject to random error and variability
- Observer variabilityObserver variability
- Instrument variabilityInstrument variability
- Subject variabilitySubject variability
Assessing PrecisionAssessing Precision
Inter and Intraobserver reproducibilityInter and Intraobserver reproducibility
Within and between instrument Within and between instrument reproducibilityreproducibility
- Continuous variables – Coefficient of Continuous variables – Coefficient of variationvariation
- Categorical – kappa statisticCategorical – kappa statistic
Enhancing PrecisionEnhancing Precision
Standardize measurement methodsStandardize measurement methods
Train and certify observersTrain and certify observers
Refining the instrumentsRefining the instruments
Automating the instrumentAutomating the instrument
Repetition (reduces random error)Repetition (reduces random error)
AccuracyAccuracy
““Does the variable actually measure or Does the variable actually measure or represent what it intends to”represent what it intends to”
Assessed by comparison to a “Gold Standard”Assessed by comparison to a “Gold Standard”
Different than precision, but many things that Different than precision, but many things that improve precision improve accuracyimprove precision improve accuracy
A function of systematic errorA function of systematic error- Observer biasObserver bias- Subject BiasSubject Bias- Instrument BiasInstrument Bias
Enhancing AccuracyEnhancing Accuracy
Standardized measurement methodsStandardized measurement methods
Training observersTraining observers
Refining instrumentsRefining instruments
Automating instrumentsAutomating instruments
Making Unobtrusive measuresMaking Unobtrusive measures
BlindingBlinding
Calibration of InstrumentsCalibration of Instruments
ValidityValidityAccuracy when there is no “Gold Standard”Accuracy when there is no “Gold Standard”
- Measuring an abstract or subjective Measuring an abstract or subjective phenomena (e.g. – pain, quality of life)phenomena (e.g. – pain, quality of life)
- Content Validity (Face, Inherent or Content Validity (Face, Inherent or sampling validity)sampling validity)
- Construct ValidityConstruct Validity
- Criterion Related Validity (Predictive Criterion Related Validity (Predictive Validity)Validity)
Final ThoughtsFinal Thoughts
An outcome measure should be sensitive An outcome measure should be sensitive enough to determine important clinical enough to determine important clinical differencesdifferencesIt should be associated with only the It should be associated with only the characteristic of interestcharacteristic of interestMeasurements should involve data Measurements should involve data collection that is efficient in time and costcollection that is efficient in time and costEfficiency is improved by increasing the Efficiency is improved by increasing the quality of each item measuredquality of each item measured