Assisting GPRA Report Assisting GPRA Report for MSPfor MSP

Xiaodong Zhang, WestatXiaodong Zhang, Westat

MSP Regional ConferenceMSP Regional Conference

Miami, January 7-9, 2008Miami, January 7-9, 2008

GPRAGPRA

GPRA measuresGPRA measuresGPRA Measure—GPRA Measure—the percentage of MSP the percentage of MSP teachers who significantly increase their content teachers who significantly increase their content knowledge, as reflected in project-level pre- and knowledge, as reflected in project-level pre- and post-assessmentspost-assessments

GPRA Measure—GPRA Measure—the percentage of MSP the percentage of MSP projects that use an experimental or quasi-projects that use an experimental or quasi-experimental design for their evaluations that are experimental design for their evaluations that are conducted successfully and that yield conducted successfully and that yield scientifically valid resultsscientifically valid results

DQI and GPRADQI and GPRA

Data Quality Initiative: Data Quality Initiative: technical assistance technical assistance to ED grant programs to improve the quality to ED grant programs to improve the quality of information on program performanceof information on program performance

GPRA Measure: MSP Teacher Content GPRA Measure: MSP Teacher Content Knowledge (MSPTCK) softwareKnowledge (MSPTCK) software

GPRA Measure: Evaluation rubricGPRA Measure: Evaluation rubric

MSPTCK objectivesMSPTCK objectives

Designed to assist grantees to determine the number of Designed to assist grantees to determine the number of teachers who have made significant gains in content teachers who have made significant gains in content knowledge (APR Q8b-f)knowledge (APR Q8b-f)

MathematicsMathematics b) Number of teachers with both pretest and posttest in math b) Number of teachers with both pretest and posttest in math

content knowledgecontent knowledge c) Number of teachers who showed significant gains in math c) Number of teachers who showed significant gains in math

content knowledgecontent knowledge

ScienceScience e) Number of teachers with both pretest and posttest in science e) Number of teachers with both pretest and posttest in science

content knowledgecontent knowledge f)f) Number of teachers who showed significant gains in science Number of teachers who showed significant gains in science

content knowledgecontent knowledge

MSPTCK processMSPTCK process

Two-step processTwo-step process

Step 1. Determine if overall posttest scores are Step 1. Determine if overall posttest scores are significantly higher than pretest (paired-samples significantly higher than pretest (paired-samples t-test)t-test)

Step 2. Determine the number of teachers in the Step 2. Determine the number of teachers in the tested group who made significant gainstested group who made significant gains If more than one test used for different groups, run If more than one test used for different groups, run

application separately for each testapplication separately for each test If more than one test used for the same group, select If more than one test used for the same group, select

the most relevant onethe most relevant one

MSPTCK tutorialMSPTCK tutorial

Access the spreadsheet onlineAccess the spreadsheet online

MSPTCK tutorialMSPTCK tutorial

MSPTCK tutorialMSPTCK tutorial

Enter data hereEnter data here tab tab Blank rows not allowed Blank rows not allowed

between databetween data

Import data from Import data from another Excel fileanother Excel file

Perform range check Perform range check

MSPTCK tutorialMSPTCK tutorial

MSP tutorialMSP tutorial

View results here View results here tabtab

MSPTCK tutorialMSPTCK tutorial

Information Information tabtab

Evaluation rubricEvaluation rubric

The criteria on the rubric are the minimum The criteria on the rubric are the minimum criteria that need to be met for an criteria that need to be met for an evaluation to have been successfully evaluation to have been successfully conducted and yield valid data conducted and yield valid data

An evaluation has to meet each criterion in An evaluation has to meet each criterion in order to meet the GPRA measureorder to meet the GPRA measure

Rubric: designsRubric: designs

Experimental study—Experimental study—the study measures the intervention’s effect the study measures the intervention’s effect by randomly assigning individuals (or other units, such as by randomly assigning individuals (or other units, such as classrooms or schools) to a group that participated in the classrooms or schools) to a group that participated in the intervention, or to a control group that did not, and then compares intervention, or to a control group that did not, and then compares post-intervention outcomes for the two groupspost-intervention outcomes for the two groups

Quasi-experimental study—Quasi-experimental study—the study measures the intervention’s the study measures the intervention’s effect by comparing post-intervention outcomes for treatment effect by comparing post-intervention outcomes for treatment participants with outcomes for a comparison group (that was not participants with outcomes for a comparison group (that was not exposed to the intervention), chosen through methods other than exposed to the intervention), chosen through methods other than random assignment. For example:random assignment. For example:

Comparison-group study with equatingComparison-group study with equating—a study in which statistical —a study in which statistical controls and/or matching techniques are used to make the treatment controls and/or matching techniques are used to make the treatment and comparison groups similar in their pre-intervention characteristicsand comparison groups similar in their pre-intervention characteristics

Rubric: designsRubric: designs Regression-discontinuity studyRegression-discontinuity study—a study in which individuals (or —a study in which individuals (or

other units, such as classrooms or schools) are assigned to other units, such as classrooms or schools) are assigned to treatment or comparison groups on the basis of atreatment or comparison groups on the basis of a “cutoff” score “cutoff” score on a pre-intervention non-dichotomous measureon a pre-intervention non-dichotomous measure

Other studyOther study——The study uses a design other than a The study uses a design other than a randomized controlled trial, comparison-group study with randomized controlled trial, comparison-group study with equating, or regression-discontinuity study, including equating, or regression-discontinuity study, including pre-post pre-post studies, whichstudies, which measure the intervention’s effect measure the intervention’s effect based on the pre-test to post-test differences of a single based on the pre-test to post-test differences of a single group, and comparison-group studies without equating, group, and comparison-group studies without equating, or non-experimental studies that compare outcomes of or non-experimental studies that compare outcomes of groups that vary with respect to implementation fidelity or groups that vary with respect to implementation fidelity or program dosage program dosage

Rubric: aspectsRubric: aspects

For Experimental Designs:For Experimental Designs:Sample SizeSample SizeQuality of Measurement InstrumentsQuality of Measurement InstrumentsQuality of Data Collection MethodsQuality of Data Collection MethodsAttrition Rates, Response RatesAttrition Rates, Response RatesRelevant Statistics ReportedRelevant Statistics Reported

For Quasi-experimental Designs:For Quasi-experimental Designs:All of the above, plusAll of the above, plus

Baseline Equivalence of GroupsBaseline Equivalence of Groups

Rubric: sample sizeRubric: sample size

Met the criterionMet the criterion—sample size was adequate —sample size was adequate (i.e., based on power analysis with (i.e., based on power analysis with recommended significance level=0.05, recommended significance level=0.05, power=0.8, and a minimum detectable effect power=0.8, and a minimum detectable effect informed by the literature or otherwise justified) informed by the literature or otherwise justified)

Did not meet the criterionDid not meet the criterion —the sample size —the sample size was too small was too small

Did not address the criterion Did not address the criterion

Rubric: quality of measurement Rubric: quality of measurement instrumentsinstruments

Met the criterionMet the criterion—the study used existing —the study used existing data collection instruments that had already data collection instruments that had already been deemed valid and reliable to measure been deemed valid and reliable to measure key outcomes, or data collection instruments key outcomes, or data collection instruments developed specifically for the study were developed specifically for the study were sufficiently pre-tested with subjects who were sufficiently pre-tested with subjects who were comparable to the study samplecomparable to the study sample

Did not meet the criterionDid not meet the criterion—the key data —the key data collection instruments used in the evaluation collection instruments used in the evaluation lacked evidence of validity and reliability lacked evidence of validity and reliability

Did not address the criterionDid not address the criterion

Rubric: quality of data collection Rubric: quality of data collection methodsmethods

Met the criterionMet the criterion—the methods, procedures, —the methods, procedures, and timeframes used to collect the key and timeframes used to collect the key outcome data from treatment and control outcome data from treatment and control groups groups were the samewere the same

Did not meet the Did not meet the criterioncriterion—instruments/assessments were —instruments/assessments were administered differently in manner and/or at administered differently in manner and/or at different times to treatment and control group different times to treatment and control group participantsparticipants

Rubric: data reduction ratesRubric: data reduction rates

Met the criterion—Met the criterion—(1) the study measured the key outcome (1) the study measured the key outcome variable(s) in the post-tests for at least 70% of the original study variable(s) in the post-tests for at least 70% of the original study sample (treatment and control groups combined), or there is sample (treatment and control groups combined), or there is evidence that the high rates of data reduction were unrelated to evidence that the high rates of data reduction were unrelated to the intervention, the intervention, ANDAND (2) the proportion of the original study (2) the proportion of the original study sample that was retained in follow-up data collection activities sample that was retained in follow-up data collection activities (e.g., post-intervention surveys) and/or for whom post-intervention (e.g., post-intervention surveys) and/or for whom post-intervention data were provided (e.g., test scores) was similar for both the data were provided (e.g., test scores) was similar for both the treatment and control groups (i.e., less or equal to a 15-percent treatment and control groups (i.e., less or equal to a 15-percent difference), or the proportion of the original study sample that was difference), or the proportion of the original study sample that was retained in the follow-up data collection was different for the retained in the follow-up data collection was different for the treatment and control groups, but sufficient steps were taken to treatment and control groups, but sufficient steps were taken to address this differential attrition in the statistical analysisaddress this differential attrition in the statistical analysis

Rubric: data reduction ratesRubric: data reduction rates

Did not meet the criterion—Did not meet the criterion—(1) the study failed to (1) the study failed to measure the key outcome variable(s) in the post-tests for measure the key outcome variable(s) in the post-tests for 30% or more of the original study sample (treatment and 30% or more of the original study sample (treatment and control groups combined), and there is no evidence that control groups combined), and there is no evidence that the high rates of data reduction were unrelated to the the high rates of data reduction were unrelated to the intervention; intervention; OROR (2) the proportion of study participants (2) the proportion of study participants who participated in follow-up data collection activities who participated in follow-up data collection activities (e.g., post-intervention surveys) and/or for whom post-(e.g., post-intervention surveys) and/or for whom post-intervention data were provided (e.g., test scores) was intervention data were provided (e.g., test scores) was significantly different for the treatment and control groups significantly different for the treatment and control groups (i.e., more than a 15-percent difference) and sufficient (i.e., more than a 15-percent difference) and sufficient steps to address differential attrition were not taken in steps to address differential attrition were not taken in the statistical analysisthe statistical analysis

Did not address the criterionDid not address the criterion

Rubric: relevant statistics reportedRubric: relevant statistics reported

Met the criterion—Met the criterion—the final report includes treatment the final report includes treatment and control group post-test means and tests of and control group post-test means and tests of statistical significancestatistical significance for key outcomes or provides for key outcomes or provides sufficient information for calculation of statistical sufficient information for calculation of statistical significance (e.g., mean, sample size, standard significance (e.g., mean, sample size, standard deviation/standard error)deviation/standard error)Did not meet the criterion—Did not meet the criterion—the final report does not the final report does not include treatment and control group post-test means include treatment and control group post-test means and/or tests of statistical significance for key outcomes and/or tests of statistical significance for key outcomes or provide sufficient information for calculation of or provide sufficient information for calculation of statistical significance (e.g., mean, sample size, statistical significance (e.g., mean, sample size, standard deviation/standard error) standard deviation/standard error) Did not address the criterionDid not address the criterion

Rubric: baseline equivalence of Rubric: baseline equivalence of groups for QEDgroups for QED

Met the criterion—Met the criterion—there were no significant pre-there were no significant pre-intervention differences between treatment and intervention differences between treatment and comparison group participants on variables related to comparison group participants on variables related to the study’s key outcomes, or adequate steps were the study’s key outcomes, or adequate steps were taken to address the lack of baseline equivalence in taken to address the lack of baseline equivalence in the statistical analysisthe statistical analysis

Did not meet the criterion—Did not meet the criterion—there were statistically there were statistically significant pre-intervention differences between significant pre-intervention differences between treatment and comparison group participants on treatment and comparison group participants on variables related to the study’s key outcomes, and no variables related to the study’s key outcomes, and no steps were taken to address lack of baseline steps were taken to address lack of baseline equivalence in the statistical analysisequivalence in the statistical analysis

Did not address the criterionDid not address the criterion

DisclaimerDisclaimer

The instructional practices and assessments The instructional practices and assessments discussed or shown in this presentation are discussed or shown in this presentation are not intended as an endorsement by the U.S. not intended as an endorsement by the U.S. Department of Education. Department of Education.