CREPCenter for Research in Educational Policy

The Center for Research in Educational Policy

Best Practices in Program Evaluation: Strategies for

Increasing Survey Response Rates

Presentation for the CREATE Conference

October 8, 2009 Louisville, KY

Background

• Established in 1989• State of TN Center of Excellence• Interim Director: Dr. Marty Alberg• Staff Includes:

– 23 Research/Research Support– 5 Statisticians– 8 GA/Student Workers– 12 Administrative/Accounting/

Technical Support

• Educational research and evaluation in a wide variety of areas in PK–12 education– Scientifically-based research– Program evaluation– Formative evaluation– Data collection training– Instrument development– Leadership academies

What do we do?

Who do we work with?

• Federal Government• State departments of education• Regional Education Laboratories • Higher education institutions • Evaluation organizations• School districts• Individual schools• Community-based organizations• Program developers

Project Areas

• Literacy and early literacy• Charter schools• Supplemental Educational Services• Educational technology• Teacher education and mentoring• Principal training/development• Urban school reform • Psychometrics

Supplemental Educational Services

• CREP has been involved with SES evaluations since 2004

• Conducted multi-year evaluations in 13 states

• Worked with the Center on Innovation and Improvement– Evaluating Supplemental Educational

Service Providers: Suggested Strategies for States

– Improving SES Quality State Approval, Monitoring, and Evaluation of SES Providers

Why Evaluate?

• Meet Federal Requirements:– States must remove providers from

approved list if they fail to:• Increase students’ achievement for 2

consecutive years• Provide services consistent with applicable

federal, state, and local health, safety and civil rights requirements

Why Evaluate?

• Formalize accountability system• Communicate plan, expectations and

results

• Identify Strengths and Weaknesses• Base improvement planning on

objective data• Document successes as supportive

evidence

Figure 1. Components of a Comprehensive SES/Evaluation Modeling Plan

STUDENTACHIEVEMENT

ServiceDelivery

CustomerSatisfaction

ProviderSurvey

District CoordinatorSurvey

Principal/LiaisonSurvey

Teacher Survey

Parent Survey

AdditionalTests

StateTests

Overall Provider Assessment

Possible Research Questions

Some of the questions currently used to address issues concerning:

• Provider Effectiveness• NCLB compliance• District and state level

implementation  

Possible Research Questions

• What are the effects of provider services on students in reading/language arts and mathematics?

• Do districts make SES available to eligible students?• Do schools and providers work together to integrate

services to meet the needs of eligible SES students?• Are providers communicating regularly with stakeholders?•  Are providers adapting tutoring services aligned with each

school’s curriculum and/or classroom curriculum? •  Are providers aligning curriculum with local and state

academic standards?•  Are providers offering services to students designated as

special education or English Language Learner (ELL)?•  What are the stakeholders’ (non-providers) overall

assessments of provider performance?•  What are providers’ experiences with and assessments of

SES interventions? 

Why Survey?

• Stakeholder perceptions are vital in understanding implementation

• Federal guidelines strongly encourage parental feedback

• Survey results can inform decisions when achievement results are insignificant or negligible

• Provide fuller picture of the quality of service and implementation

Paper-Based and Online Surveys

SES Evaluations: Stakeholder Feedback

• Paper-Based Survey for Parents• Online Survey for District

Coordinators• Online Survey for Principals/Site

Coordinators• Online Survey for Teachers• Online Survey for Providers

An SES Evaluation Contains:

• An Overall Statewide Assessment of SES:– Aggregated Stakeholder Results– Student Achievement Results for SES

Providers• Reading/Language Arts and Mathematics

• Individual Provider Assessments:– Stakeholder Results – Student Achievement Results

• Reading/language and Mathematics

Rubric of Overall Evaluation of Provider Effectiveness Outcome Insufficient

InformationBelow

StandardsMarginalQuality

Acceptable Above Standards

1. Student Achievement

Insufficient information (insufficient sample size; non-significant

results; or no achievement data)

Students have not shown gains related to tutoring. Results are statistically

significant and favor non-SES students

There is evidence that some tutored students are making achievement gains. Overall comparison is statistically

significant, with effect size up to +.17

There is evidence that some tutored students are making achievement gains. Overall comparison is statistically significant, with effect size ranging from +.18 to +.25

There is evidence that some tutored students are making

substantive achievement gains. Overall comparison is statistically significant, with effect size greater than +.25

2. Communication

Insufficient Information Provider communication weak or nonexistent

Provider communication inconsistent

Provider is adequately communicating with key

stakeholders.

Provider regularly and frequently communicates with

key stakeholders.

3. InstructionalPlans

Insufficient Information Instructional plans not geared to student needs or

reinforcement of regular academic program

Provider inconsistently planned instruction geared to

student needs or reinforcement of

regular academic program

Provider made attempts to plan instruction geared to student

needs or reinforcement of regular academic program

Provider instructional plans geared to student needs or

reinforcement of regular academic program

4. Local and State Standards

Insufficient Information Provider services not in alignment with local and state

academic standards

Provider services inconsistently aligned with local and state academic

standards

Provider services sometimes aligned with local and state

academic standards

Provider services in alignment with local and state academic

standards

5. Special Education and ELL Students

Insufficient Information Provider did not offer accommodations to special education or ELL students

Provider inconsistently offered accommodations to special education or ELL students

Provider sometimes offered accommodations to special education or ELL students

Provider offered accommodations to special education or ELL students.

6. Assessment of Provider Overall

Insufficient Information Dissatisfaction with provider overall

Inconsistent satisfaction with the provider overall

Some satisfaction with provider overall

Satisfaction with provider overall

Parent Paper Survey Distribution

• During the 2008-2009 school year:– Over 31,000 paper parent surveys were

distributed in 9 states– In 8 states, surveys were printed in

English on one side and Spanish on the other side

– Surveys were packaged and sent in bundles to SES directors to deliver to SES schools

Parent Paper Survey Distribution

• Traditional Distribution Method:1. Surveys with instructions are printed,

packaged and sent to SES districts2. Between 40 and 60 survey packets are sent,

boxed, for the district coordinator to deliver to SES schools

3. Principals/site coordinators distribute to SES students

4. Students bring them home, parents complete them.

5. Students bring them back to school.6. After several weeks, principals/site

coordinators mail surveys to CREP in postage-paid envelopes provided by CREP

Parent Paper Survey Distribution

• During the 2008-2009 school year, 2 states opted for a different delivery mode: – One district chose to directly mail

surveys to the homes of a sample of parents– the other districts in the state used the traditional method

– One state opted for district coordinators to directly address packet envelopes with parents’ names, deliver packets to school with cover letter from the district, and collect surveys from schools

2008-2009 Parent Response Rates

• Overall response rates for 2008-2009, based on number of surveys sent to schools:– Range was from 8% to 38%– Median of 17%– An increase from previous year’s

median (12%)

Paper Survey Distribution

• States which distributed traditionally:– Response rates ranged from 10% to

33%– The median rate was 17%

• The district that directly mailed to a sample of parents:– Response rate was 8%; overall response

rate for the state was 11%• The state in which surveys were

addressed to parents and delivered by district coordinators had a response rate of 38%

Challenges in Reaching Parents• Timing

• Tight survey window due to standardized testing • Contractual agreements can delay survey process

• Distribution• Communicating with schools and districts can be

challenging

• Difficulty in determining number of parents • Can be hard to determine how many students were

served at each school prior to mailing surveys• Response rates may change once achievement data

is received

Challenges in Reaching Parents

• Lessons Learned:– Communication is the key

• Between Evaluator and State and Districts • Between State and Districts• Between Districts and Schools

– Pre-coding information, if possible, may increase response rate

– Involvement of district coordinator is crucial

– Earlier distribution is better

Challenges in Reaching Parents

For Consideration…• Sample surveys for non-participating

parents• Student surveys• Focus groups• Online surveys for parents

Online Survey Distribution

• Online Surveys are Utilized for other SES stakeholders

• During the 2008-2009 school year, CREP disseminated login information for :– 9 SES state directors – 491 SES providers– 204 SES district coordinators– 588 SES school personnel (sent to

district coordinators)

Online Survey Distribution

• In order to access the online system, user identification codes and passwords are needed

• Log in Information is sent in the spring via email

• Test emails are sent to:– Verify correct participant and email

address– Introduce and inform participant of the

study

Online Survey Distribution

• The log in information is sent to:– State SES directors – SES Providers– SES District Coordinators

• The log in information for school personnel is sent to district coordinators to forward to the SES schools

– Periodic reminders are sent in the weeks following

2008-2009 Online Response Rates

• Response rates vary among stakeholder groups:– SES Providers: Ranged from 45% to 100%

• Median of 79%– SES District Coordinators: Ranged from 47% to 100%

• Median of 79% – SES Principals/Site Coordinators: Ranged from 15% to

88%• Median of 35%

– SES Teachers: Responses representative of 7% to 50% of SES schools• Median of 24%

Online Survey Response Rates

• Wide variances between stakeholder groups– Providers and District Coordinators

most likely to respond– Principals/Site coordinators and

teachers less likely:• Communication goes through district• Have less time

Challenges in Online Response Rates

• Lessons Learned:– Communication is the key

• Between Evaluator and Districts and Schools

– Reminders are essential– Involvement of district coordinator is

crucial– Earlier distribution of login information

is better• Would allow for reminders to school personnel• Would allow feedback to state and district

coordinators concerning lack of representation from school personnel

The Process of Evaluating SES is Most Effective when:• Districts and School personnel are

invested in the process• Communication between all

stakeholders is strong and meaningful• Feedback regarding the impact of

providers is timely and understandable