1-LimitationsofRCTs2-TheEvidenceBaseinPractice3-ReadingResearchLiterature

UNMEvaluationLabSummerInstitute2018MelissaBinder

mbinder@unm.edu /evallab.unm.edu

Limitations ofRCTs

-Hugelyexpensive(sothatveryfewhappen)-HardtodowhenNissmall(ruralinterventions)- Blackbox– youdon’tknowWHYitworked,orwhichelementsarenecessary...- Ethicalconsiderations(randomizedassignmentmeansthat youcan’ttriageto helpthemostvulnerable,andthatyouwouldhavetodenyservicetosome)-Unclearwhetherresultswilltranslatetodifferentpopulation-Maystifleinnovationandhome-grownprograms

7/30/18 UNMEvaluationLabSummerInstitute- TheEvidenceBase 2

Theevidencebaseinpractice

• RFPsarelooseningupabit,askingforevidence-informed programs.

• Maybemodelledafterevidence-basedprograms,withkeyfeaturesreplicated.

• Maybebasedonevolvingunderstandingofhumanbehavior.

7/30/18 UNMEvaluationLabSummerInstitute- TheEvidenceBase 3

Implementation

iskey

StateoftheArt

Synthesizescurrentknowledge,currentunderstandingofafield.

See ApplyingtheScienceofChildDevelopmentinChildWelfareSystems inReadingsfolder.Linksnewerunderstandingoftraumatoprogrammaticapproaches.

7/30/18 UNMEvaluationLabSummerInstitute- TheEvidenceBase 5

Paradigmchanges

UnderstandingoftraumaACESstudy– Felliti etal.1998– showedassociation,notcausality

7/30/18 UNMEvaluationLabSummerInstitute- TheEvidenceBase 6

Source:Felitti etal.1998,table17/30/18 UNMEvaluationLabSummerInstitute- TheEvidenceBase 7

Table1.Prevalenceofchildhoodexposuretoabuseandhouseholddysfunction,continued

Source:Felitti etal.1998,table1

CategoryPrevalence(%)Prevalence(%)

7/30/18 UNMEvaluationLabSummerInstitute- TheEvidenceBase 8

Oddsratiosforpeoplereporting4ormoreACEs,comparedwiththosereportingnone

Basedonmedicalrecord

Heartdisease 2.2

Anycancer 1.9

Stroke 2.4

Chronicbronchitisoremphysema 3.9

Diabetes 1.6

Self-reported

Everhadhepatitisorjaundice 2.4

Self-ratedhealthfairorpoor 2.2

Source:Felitti etal.1998,tables7and8.7/30/18 UNMEvaluationLabSummerInstitute- TheEvidenceBase 9

Discussion

IstheretensionbetweenclinicalevidenceandRCTevidence?Orsynergy?

7/30/18 UNMEvaluationLabSummerInstitute- TheEvidenceBase 10

Helpfultoread

• Keyliteraturethatorganizationfindsfoundational• Keystudiesinthefield• Historicaltrauma• Earlychildhood:NFP,PerryIntervention

UNMLibraryAccess

OneuseraccountperorganizationAccessuntilJuly31,2019Searchingforarticlesathttps://library.unm.eduIfUNMdoesn’thaveanarticle,wecangetitforyouthroughinter-libraryloan

Guidetoreadingresearchstudies

Day1– ChecklistforProgramEvaluationStudiesSummerInstituteMaterials

UNMPublicPolicySeminarandPolicy&EvaluationLab:Checklistforprogramevaluationstudies

ABSTRACT RELEVANTDETAILS DESCRIPTIONOFPREVIOUSRESEARCHWhatisresearchquestion? Sample LotsofstudiessupportfindingsWhatweremainfindings? Samplesize #recruited,finalsamplesize Mixedfindings

Issamplerepresentative? yesorno,&whoisrepresented? AuthorsdistinguishqualityofstudiesDescribesample: . Authorsidentifydifficulties

Methodologytypes: Whichstudiesareworthreading? Why?RandomizedControlledTrial(RCT)Quasi-experimental

Differenceindifferences(DID)Regressiondiscontinuity(RD)InstrumentalVariables(IV)Other:describe: ProgramDetails

Observational NameofprogramBasicmultivariate(controllingforcovariates) ProgramgoalsCasestudy Theoryofchange WhyprogrammightworkOther:describe: Programdescription:

HOWCONVINCINGISSTUDY?RandomizedControlledTrial(RCT) Quasi-experimental Observational

isdatafromalargenationallyrepresentativesurvey,andifso,namethesurvey

Providespecifics:

Alwaysgoingtohavedoubtaboutrandomnessofvariation.

Suggestive:Strongassociationsestablishedandpotentialendogeneitymakesunder-estimationmorelikelythanover-estimation.(E.g.,ACEsstudy)

Problematic:Endogeneitylikely,authors'argumentstothecontraryarenotconvincing.Otherstudyflaws.

Amazing:Largeeffectsonmultiplevariablesofinterest.Programcanbescaledup.

Convincing:Smalltomoderateandstatisticallysignficanteffectsonmultiplevariablesofinterest.Maybehardtoscaleup.

Suggestive:Smalltomoderateandmostlystatisticallysignficanteffectsforafewkeyoutcomes.Maybehardtoscaleup.

Problematic:Afewsmallstatisticallysignficanteffectsoutofmanyvariablesmeasured.Noeffectsornegativeeffects.Samplepopulationisnotrepresentativeoftargetgroup.Otherstudyflaws.

Providespecifics:

Alwaysgoingtohavedoubtaboutrandomnessoftreatment.Considerwaysthatparticipantsself-selected,andwhetherself-selectionwouldlikelyover-orunder-estimateeffects.InstrumentalVariables(IV)cansometimesconvincinglybreakendogeneityconcerns.Beconcernedaboutinstrumentsthatarenothighlycorrelatedwithvariableofinterest,orthatareunlikelytobetrulyexogenous.

Convincing:Self-selectionunlikelyandtreatmentarguablyrandom.Greatinstruments.Moderatetolargestatisticallysignificanteffects.Effectsarerobust,meaningtheyarestablefordifferentreasonablespecifications(e.g.,differentyears).Suggestive:Self-selectionunlikelyandtreatmentarguablyrandom.Smalltomoderatestatisticallysignficanteffects,robusttoreasonablealternativemodelspecifications.

Problematic:Self-selectionlikelyortreatmentplausiblynon-random.Estimatesaresensitivetoreasonablealternativemodelspecifications.Otherstudyflaws.

Providespecifics:

Questions?