Upload
bruce-stevens
View
219
Download
0
Tags:
Embed Size (px)
Citation preview
What Works Clearinghouse
Susan Sanchez Institute of Education Sciences
Purpose: To promote education decision making through a web-based dissemination system featuring comprehensive, systematic, high-quality reviews of studies on the effectiveness of educational interventions (programs, products, practices and policies).
The WWC Does Not:
Endorse educational interventions
Conduct field studies of the effects of interventions
Rather, the WWC reports on the effectiveness of educational
interventions as measured by available evidence.
Challenges
Research literature too abundant for individual selection. Where to begin?
Difficult to sift and know what to trust and what to use. Lots of claims of effectiveness out there, but what can you trust?
It has to be transparent if it’s to be believed.
Complex, technical issues can take more than forever and ever to resolve.
Challenges
Opps, we got the answer but now there are new advances in methodology.
Consumers need a fast response for evidence on what works.
Pretty wonky stuff. How do you create user-friendly web-based reports and products that meets your customers’ needs?
WWC Systematic Review Process
A systematic review is a review of the evidence on a clearly formulated question that uses systematic and explicit methods to identify, select and critically appraise relevant research, and to extract and analyze data from the studies that are included in the review.
WWC Systematic Review Process
Select the Topic Develop the review protocol
Research questions Search parameters Inclusion/Exclusion criteria
Conduct a literature search & identify the research
Screen and code studies for relevance and methodological quality
Analyze the results of eligible studies Summarize results
Study Review Process
Does Not Pass Screen
Meets WWC Evidence Standards• Meets WWC Evidence Standards•Meets WWC Evidence Standards
with Reservations
Submissions from the Public & Intervention
Developers
Literature Searches
Screening Standards
Pass Screen
Study Reviewed Against the WWC Evidence Standards
Does Not Meet WWC Evidence Screens
Develop ProtocolDevelop Protocol
Three Stages of Review
Screen studies for relevance Select only relevant evidence
Assessing strength of the evidence and sort by rigor Select only credible evidence Sort credible evidence by whether any
reservations
Identify other important study characteristics
Stage 1: Screening for Relevance
All potentially relevant studies identified through the extensive search
Does the study provide evidence that, if credible, would be a valuable addition to the knowledge base regarding the effectiveness of the focal intervention?
Six Essential Screeners
1) Relevant time frame
2) Relevant intervention
3) Relevant sample
4) Relevant outcome
5) Adequate outcome measures
6) Adequate reporting
Stage 2: Assessing Strength of Evidence
The goal is to apply consistent criteria to sort evidence by its credibility into three buckets:
Meets Evidence Standards
Meets Evidence Standards with Reservations
Does Not Meet Evidence Screens
The WWC Evidence Standards (applied to individual studies)
• Meets Evidence Standards
• RCTs without severe design or implementation flaws
• Meets Evidence Standards with Reservations
• RCTs with severe design or implementation flaws
• QEDs with equating and without severe design or implementation flaws
• Does Not Meet Evidence Screens
•Studies not relevant to review
•Studies with fatal design or implementation flaws
Five Factors Govern Judgments
Method for forming intervention and comparison groups
Evidence on baseline equivalence
Sample attrition
Possible contamination of study conditions
Threat of teacher confound
Stage 3: Other Study Characteristics
Variations in people, settings and outcomes represented How generalizable are the findings?
Results reported by subgroups, settings and outcomes What is the breadth of the outcomes
reported?
Statistical results available How complete is the reporting? Do the estimates reflect statistical
controls for baseline characteristics?
Evidence Base of Character Education Programs
Over 70 programs were submitted to or identified by the WWC. 41 school-based programs met WWC definition and were eligible for review. 93 studies on the 41 programs were collected.
13 programs had at least one study meeting evidence standards, either with or without reservations
27 programs had no studies passing evidence screens 1 program is under review
13 Character Education Programs with at Least one Study Meeting Evidence
Standards, either with or without Reservations
1. An Ethics Curriculum for Children 8. Skills for Action
2. Building Decision Skills 9. Skills for Adolescence
3. Caring School Communities 10. Too Good for Drugs
4. Connect with Kids 11. Too Good for Violence
5. Facing History and Ourselves 12. Too Good for Violence and Drugs
6. Lessons in Character7. Positive Action
13. Voices Literature and Character Education Program
Distribution of 92 Studies on 40 Programs
across Evidence Categories7 (8%)
11 (12%)
74 (80%)
Meet evidence standards with reservations
Meet evidence standards
Do not meet evidence screens
Common Reasons for Studies Failing to Meet Evidence
Screens
Not meet relevance screens Lack of a comparison group Severe overall or differential attrition for QEDs Lack of baseline equivalence for QEDs Confound in one-teacher/school-per-condition studies Inadequate statistical reporting for ES computation
For More Information on the WWC
• Visit our website at www.whatworks.ed.gov
• Subscribe for WWCUpdate, the WWC’s electronic news alert through our website
• Phone: 1-866-WWC-9799 E-mail: [email protected]