What Works Clearinghouse Susan Sanchez Institute of Education Sciences

What Works Clearinghouse

Susan Sanchez Institute of Education Sciences

Purpose: To promote education decision making through a web-based dissemination system featuring comprehensive, systematic, high-quality reviews of studies on the effectiveness of educational interventions (programs, products, practices and policies).

The WWC Does Not:

Endorse educational interventions

Conduct field studies of the effects of interventions

Rather, the WWC reports on the effectiveness of educational

interventions as measured by available evidence.

Challenges

Research literature too abundant for individual selection. Where to begin?

Difficult to sift and know what to trust and what to use. Lots of claims of effectiveness out there, but what can you trust?

It has to be transparent if it’s to be believed.

Complex, technical issues can take more than forever and ever to resolve.

Challenges

Opps, we got the answer but now there are new advances in methodology.

Consumers need a fast response for evidence on what works.

Pretty wonky stuff. How do you create user-friendly web-based reports and products that meets your customers’ needs?

WWC Systematic Review Process

A systematic review is a review of the evidence on a clearly formulated question that uses systematic and explicit methods to identify, select and critically appraise relevant research, and to extract and analyze data from the studies that are included in the review.

WWC Systematic Review Process

Select the Topic Develop the review protocol

Research questions Search parameters Inclusion/Exclusion criteria

Conduct a literature search & identify the research

Screen and code studies for relevance and methodological quality

Analyze the results of eligible studies Summarize results

Study Review Process

Does Not Pass Screen

Meets WWC Evidence Standards• Meets WWC Evidence Standards•Meets WWC Evidence Standards

with Reservations

Submissions from the Public & Intervention

Developers

Literature Searches

Screening Standards

Pass Screen

Study Reviewed Against the WWC Evidence Standards

Does Not Meet WWC Evidence Screens

Develop ProtocolDevelop Protocol

Three Stages of Review

Screen studies for relevance Select only relevant evidence

Assessing strength of the evidence and sort by rigor Select only credible evidence Sort credible evidence by whether any

reservations

Identify other important study characteristics

Stage 1: Screening for Relevance

All potentially relevant studies identified through the extensive search

Does the study provide evidence that, if credible, would be a valuable addition to the knowledge base regarding the effectiveness of the focal intervention?

Six Essential Screeners

1) Relevant time frame

2) Relevant intervention

3) Relevant sample

4) Relevant outcome

5) Adequate outcome measures

6) Adequate reporting

Stage 2: Assessing Strength of Evidence

The goal is to apply consistent criteria to sort evidence by its credibility into three buckets:

Meets Evidence Standards

Meets Evidence Standards with Reservations

Does Not Meet Evidence Screens

The WWC Evidence Standards (applied to individual studies)

• Meets Evidence Standards

• RCTs without severe design or implementation flaws

• Meets Evidence Standards with Reservations

• RCTs with severe design or implementation flaws

• QEDs with equating and without severe design or implementation flaws

• Does Not Meet Evidence Screens

•Studies not relevant to review

•Studies with fatal design or implementation flaws

Five Factors Govern Judgments

Method for forming intervention and comparison groups

Evidence on baseline equivalence

Sample attrition

Possible contamination of study conditions

Threat of teacher confound

Stage 3: Other Study Characteristics

Variations in people, settings and outcomes represented How generalizable are the findings?

Results reported by subgroups, settings and outcomes What is the breadth of the outcomes

reported?

Statistical results available How complete is the reporting? Do the estimates reflect statistical

controls for baseline characteristics?

Evidence Base of Character Education Programs

Over 70 programs were submitted to or identified by the WWC. 41 school-based programs met WWC definition and were eligible for review. 93 studies on the 41 programs were collected.

13 programs had at least one study meeting evidence standards, either with or without reservations

27 programs had no studies passing evidence screens 1 program is under review

13 Character Education Programs with at Least one Study Meeting Evidence

Standards, either with or without Reservations

1. An Ethics Curriculum for Children 8. Skills for Action

2. Building Decision Skills 9. Skills for Adolescence

3. Caring School Communities 10. Too Good for Drugs

4. Connect with Kids 11. Too Good for Violence

5. Facing History and Ourselves 12. Too Good for Violence and Drugs

6. Lessons in Character7. Positive Action

13. Voices Literature and Character Education Program

Distribution of 92 Studies on 40 Programs

across Evidence Categories7 (8%)

11 (12%)

74 (80%)

Meet evidence standards with reservations

Meet evidence standards

Do not meet evidence screens

Common Reasons for Studies Failing to Meet Evidence

Screens

Not meet relevance screens Lack of a comparison group Severe overall or differential attrition for QEDs Lack of baseline equivalence for QEDs Confound in one-teacher/school-per-condition studies Inadequate statistical reporting for ES computation

For More Information on the WWC

• Visit our website at www.whatworks.ed.gov

• Subscribe for WWCUpdate, the WWC’s electronic news alert through our website

• Phone: 1-866-WWC-9799 E-mail: info@whatworks.ed.gov