PTP 560

• Research Methods

Week 6

Thomas Ruediger, PT

SamplingHow can we generalize our study to the world?

Our sample responses are representative of population!

• Population– All members– All measurements possible

• Sample– Subgroup of members– Measurements actually taken

• Bias– Conscious– Unconscious (sub?)

Sampling Target PopulationAccessible population Sample

• Inclusion Criteria– Trait of the Target or accessible population– Qualifies someone as a subject– Restrictions here will limit ability to generalize

• Exclusion Criteria– Precludes someone being a subject– Excluded because they may interfere with interpreting findings

• Selection– Plan – Fig 8.2

Sampling Techniques

• Probability– Through random selection– Not the same as random assignment– Every member has equal chance of being selected– Considered (but not guaranteed to be) representative– Allows estimate of sampling error

The difference between population average and sample average

• Non-probability– Non-random methods– Limits ability to generalize outcomes

Probability Sampling

• Simple random– Also known as sampling without replacement– Table of random numbers (Table 8.1)

• Systematic

• Stratified random– Subsets (strata) established– Random selection from the strata– May also be proportional– May be more representative than random

Probability Sampling

• Disproportional– Select random samples of appropriate size– Correct it with proportional weighting

• Cluster– Successive random sampling– Convenient and efficient…………….BUT, increased sampling error

– Example• Area probability sampling• Random digit dialing

Non-Probability Sampling• Convenience– Also known as accidental sample– Consecutive sampling is common method– Self selection is a major limitation

• Quota– Enroll subjects– Stop for certain strata when they are represented

• Purposive– Hand picked by criteria– Prone to bias

• Snowball– Chain-referral

Recruitment• Feasibility issues can be daunting• Advertisements• Other healthcare providers/institutions• Track and report– Screened for eligibility– Number actually eligible– Number enrolled

• POWER– “The ability to find significant differences when they exist”– Important to know a priori to get appropriate sample size

Validity in Experimental Design

• Experiment has three essential characteristics:1. Manipulation of independent variables

2. Random assignment to groups

3. Control or comparison group

Supports (Does NOT prove) cause-and effect relationship

• Extraneous variables– Must be controlled OR,– They can confound

`

Handling incomplete (or lost) data

• On-Protocol (Completer) Analysis– Only those who complete the study– Tends to bias in favor of the treatment

• Intentions to treat (ITT) PREFERRED approach– What did we intend to do?– More conservative than On-Protocol– Considered to reflect clinical situations– Analysis?

• Non-completer equals failure• Last observation carried forward

Validity in Experimental Design

• Blinding– Single Blind

• Subject blinded to treatment or placebo

– Double Blind• Subject and Tester blinded to treatment or placebo

condition

– Triple Blind• Researcher, tester, and subject blinded• Data analyzed by independent source

Controlling Inter-subject Differences

• Options– Homogenize on certain characteristic(s)–Manipulate attribute variables into “Blocks”– Consider matching– Use subjects as own control– Handle statistically with ANCOVA

• Table 9.1

Threats to validityFour Threats correspond to four major questions

• Is there a relationship between IV and DV?– Statistical Conclusion Validity

• Evidence of causal relationship?– Internal Validity

• Can results be generalized to a theoretical construct?– Construct Validity

• Can it be generalized to other persons/settings/times?– External Validity

Statistical Conclusion Validity

• Is there a relationship between IV and DV?

• Threats– Low Statistical Power– Violated Assumptions of Statistical Tests– Error Rate– Reliability– Variance– Failure to use ITT

Internal Validity• Evidence of causal relationship?The extent to which the results of a study/experiment can be attributed

to the treatment or intervention rather than to flaws in the research design

• Threats to internal validity– History– Maturation– Attrition– Testing– Instrumentation– Regression– Social Threats

Internal Validity

• Threats to Internal Validity – Testing Interactions• Pre-tests or subsequent testing has an effect• Second test scores tends to move toward the mean• Standard Deviation decreases

Construct Validity

• Can results be generalized to a theoretical construct?

• Threats– Limits of Operational Definitions– Time Frame Within Operational Definitions– Multiple Treatment Interactions– Experimental Bias– Hawthorne Effect

External Validity

• Can results be generalized to other persons/settings/times?

• Threats– Interaction of treatment and selection– Interaction of treatment and setting– Interaction of treatment and history

Research Designs

• Common Sources of Error – Experimental Bias• Post Hoc Error

– Events that occur in sequence without cause & effect– Change related to coincidence; rival hypothesis

• Error of Misplaced Precision– Statistical significance not clinically important– Measuring blood pressure to the 0.001 mm Hg

Research Designs

• Common Sources of Error – Experimental Bias• “Typical” Case Studies

– Typically not typical – Typically IDEAL

• The Law of the Instrument– Always use the same instrument– Always calibrate the instrument

Experimental Bias

• Halo Effect– Irrelevant factors effect outcome favorably or

unfavorably– Ex: Health care worker with a favorable/unfavorable

characteristic influences outcome of study

• Rating Errors–Over/Under/Central tendency raters

• Hawthorne Effect– 1920’s Hawthorne Plant of Western Electric– Productivity & efficiency

Experimental Bias

• “Self-Fulfilling Prophecy”– Find what researchers expect to find

• “John Henry” Effect–Control group discovers their status and

outperforms experimental group

• Placebo Effect– True effect of intervention versus ‘suggestibility’