Strategies for Strategies for Effective Program Effective Program

EvaluationsEvaluations

U.S. Department of U.S. Department of EducationEducation

The contents of this presentation were produced by the Coalition for Evidence-Based Policy, in partnership with the National Opinion Research Center (NORC) at the University of Chicago, under a contract with the Institute of Education Sciences.

Step 1: Step 1: Find a researcher with expertise Find a researcher with expertise in conducting rigorous impact in conducting rigorous impact evaluations evaluations to include on the to include on the

study team.study team. Contact authors of previous, well-designed impact evaluations.

Ask the evaluator to identify a plan that describes in non-technical language, how they would conduct the study.

Check references

Step 2: Step 2: Decide what research Decide what research question(s) the study seeks question(s) the study seeks to answerto answer..

Evaluate a specific, well-defined MSP approach.

Measure the effect of the program on teachers’ content knowledge.

Measure the impact on student knowledge.

If resources permit, ask questions about the long-term effect of the MSP projects.

Step 3: Step 3:

Decide on the study design.Decide on the study design.

 

Decide on overall design: preferably a randomized controlled trial, or if not possible, a well-matched comparison-group study.

Decide the sample size to measure teacher content knowledge – at least 90 teachers needed.

Decide the sample size to measure student achievement – at least 60 teachers needed.

Step 3 continued: Step 3 continued:

 

Decide how to recruit and allocate teachers.

The simplest way to allocate teachers in a RCT is to apply random assignment to the entire sample. However, if there are big differences across the group (high-achieving, low-achieving) then grouping in blocks and then random assignment.

In a comparison group study the teachers must be very closely matched: First, teachers must be closely matched by achievement levels; and second, by students’ achievement.

Step 3 continued: Step 3 continued:

 

Decide how to measure MSP project outcomes for student achievement. There are 3 conditions:

--must obtain scores for individual students;--must obtain scores before they enter the teachers class, and at the end of the study; and--must be able to convert the scores so that they will enable comparisons across grade levels.

Decide how to measure teacher content knowledge.

Step 3 Example: Step 3 Example:

 

Program group

Control group

Step 4: Step 4: Gain the cooperation of Gain the cooperation of

teachers and school officialsteachers and school officials.. Identify one or more senior-level advocates for the study.

Gain teachers’ cooperation by explaining the benefits of the research design.

Satisfy privacy (FERPA) concerns in your design.

Special consideration for matched comparison group of teachers from another school district.

Step 5:Step 5:Allocate teachers to the program Allocate teachers to the program and control (or matched and control (or matched comparison) groups.comparison) groups.

The evaluator should conduct random assignment of teachers rather than project director.

Ask participating teachers not to share their materials.

Ensure that students are assigned to classes using the normal procedures.

Step 6: Step 6: Collect the data needed to Collect the data needed to measure the MSP project’s measure the MSP project’s effectiveness.effectiveness.

Make process as short and streamlined as possible.

Provide a unique personal identifier for each participant.

At the time of assignment, obtain pre-program performance data.

Make every effort to collect data from 80 percent of the original sample of teachers.

Step 7: Step 7: Analyze and report the Analyze and report the study’s results.study’s results.

Obtain regression-adjusted estimates of the MSP project’s effect on student achievement.

Use the students’ pre-program test score as a covariate.

Use data from all of the original participants.

Improved Intervention

Improved Evaluation

Help DeskHelp Desk

Website: Website: http://www.whatworkshelpdesk.ed.govhttp://www.whatworkshelpdesk.ed.gov

Phone: 1-866-WWC-9799 (8am-8pm ET Phone: 1-866-WWC-9799 (8am-8pm ET Mon-Fri) Mon-Fri)

Email: Email: info@whatworkshelpdesk.ed.govinfo@whatworkshelpdesk.ed.gov

Mission is to provide federal, state, and local education officials, researchers, program providers, and educators with practical, easy-to-use tools to (i) advance rigorous evaluations of educational interventions (i.e., programs, products, practices, and policies), and (ii) identify and implement evidence-based interventions.