ED PACE Educational Development for Planning and Conducting Evaluations West Virginia Department of Education

ED PACEED PACE

Educational Development for Planning and Conducting

Evaluations

West Virginia Department of Education

Project DesignProject Design

GoalsGoals

Provide empirical data on student achievement in virtual foreign language courses using a quasi-experimental design with experimental elements

Develop a framework for scientifically based research that can be used at the local, state, and national level to measure the impact of technology enhanced programs on student achievement

DescriptionDescription

ED PACE will generate and implement three research models and methods that can be replicated in other settings:

a summative research modela formative research modela training model

See revised project outputs chart

FOR MORE INFO...

Summative Research ModelSummative Research Model Research Question

– Does participation in program lead to student achievement gains?

Methodology – Quasi-experimental

Outcome Measures– Individual achievement data from WVEIS*

FOR MORE INFO...

*West Virginia Education Information System (WVEIS)

Sample StudiedSample Studied

Treatment Group Randomized Treatment Control Group I Control Group II* Control Group III*

FOR MORE INFO...

*see revised sample and data collection chart

Outcome MeasuresOutcome Measures

A state standardized test – WESTEST (2004-2006)– SAT9 (2001-2003)

A rubric-scored writing assessment – 7th and 10th grade only

UNIACT Interest Inventory– ACT Plan (8th)– ACT Explore (10th)

Language Acquisition Assessment Spanish I

Analytic MethodsAnalytic Methods

Step 1) Analysis of Students’ Outcomes

• Comparison of Means and Mean Gains (randomized groups)

• Regression Analysis (non-randomized groups)- Control for students’ past achievement- Control for schools’ past achievement

(controls for current year non-program education when appropriate)

Analytic MethodsAnalytic Methods

Step 2) Explore Controls to Generate “Matching Criteria”

• Goal: provide state with set of school and student-level variables that are related to outcomes.

• Process: - Analyze other student and school-level

variables (OLS and HLM) - Determine extent to which they are

subsumed within model’s current controls / account for additional variation

Formative Research Model Formative Research Model

Research Question:– How is the program implemented in different

settings? Methodology

– Documentation of full and sub-sample of treatment group over three year period

Sample– Year 1: all sites with treatment (27 schools)– Year 2: new sites and 5 focus sites (multiple visits)– Year 3: new sites and 5 focus sites (3 repeat, 2 new)

Formative ResearchFormative Research

Goals:– Provide feedback for virtual both Spanish 1A/1B

courses and Virtual School courses in general about what elements lead to successful implementation

– Inform the summative analysis of achievement data

Measures– Observation protocol– Interview protocol (principal/teacher/parent)– Surveys (student/teacher)

Training ModelTraining Model

Goal:– Transfer models and methods developed

during summative and formative research to state and local practitioners

MethodologyMentor-Novice model to develop training

materials– Year One: shadow researchers– Year Two: develop materials for training and pilot– Year Three: Implement training with local entity*

FOR MORE INFO...

* efforts will be made to extend training model to other states

InterventionIntervention

Spanish ProjectSpanish Project

West Virginia Board of Education Policy 2510 required the offering of foreign language at the 7th and 8th grade levels by the fall of 2002.

In response to requests from numerous school systems, the West Virginia Virtual School (WVVS) submitted and received a grant from the United States Department of Education to develop and deliver via Internet technology a middle school Spanish course.

Spanish DeliverySpanish Delivery

The course is delivered in a two–year split model option of Spanish I.

Spanish IA (7th grade) covers approximately one-half of the content standards for Spanish I.

Students who successfully complete Spanish IA enroll in Spanish IB (8th grade)

Upon completion of both years of study, students who have passed the courses will be awarded one (1) high school credit.

A Small Class at Paw PawA Small Class at Paw Paw

A Large Class at Warm A Large Class at Warm SpringsSprings

Classroom VisitsClassroom Visits

Time for Time for El teléfonoEl teléfono

Handbook – example of numerous Handbook – example of numerous hands-on activitieshands-on activities

Los CuadernosLos Cuadernos – 4 Modules – 4 Modules

Spanish 1A/1B 2003-2004Spanish 1A/1B 2003-2004

Calendaronline lesson plansinteractive activities

Announcements

Virtual School Internet SiteGrammarGradesQuizzes

Wimba voice Tools 8 Module Workbooks

2003-200444classes

525 students33 facilitators

14 AdjunctsGrade 30 studentsEmail 30 students

2 full time teachersWeekly phone contact

Daily Lesson Plans

ImplementationImplementation

ED PACE TeamED PACE Team

The West Virginia Department of Education will draw on several resources both within and outside of the department:

– Rockman et al – The EdVenture Group– WVEIS– Virtual School

– SETDA

FOR MORE INFO...

See related links

Management PlanManagement Plan

Granting Agency & PartnerUnited States Department of Education

OESE

Fiscal AgentProject Director

Virginia Department of EducationOffice of Educational Technology

WVEISData System

Virtual SchoolIntervention

SETDANational Dissemination

Summative Research

Training ModelEdVenture Group

Formative Research

Research TeamPrincipal Investigator

Rockman et al

FOR MORE INFO...

See organization plan with reporting schedule

Schedule: Year OneSchedule: Year One

Gather historical and current data on program

FOR MORE INFO...

Visit project calendar at: http://wvde.state.wv.us/calendar

Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug

Classroom ObservationsClassroom Observations

WVEIS Request 2003-2004 WVEIS Request 2003-2004

Develop InstrumentsDevelop Instruments

WVEIS request 2002-2003 WVEIS request 2002-2003

Analyze 2003-2004 dataAnalyze 2003-2004 data

Observe researchers Observe researchers

Gather and Clean data

Analyze data

Schedule: Year TwoSchedule: Year Two

Track short and long-term gains and facilitate transfer to state

FOR MORE INFO...



Student ObservationsStudent Observations


Develop InstrumentsDevelop Instruments

Analysis 2003-2004 Analysis 2003-2004

Analyze 2004-2005 dataAnalyze 2004-2005 data

Develop training model Develop training model Collect observation data Collect observation data

Gather and Clean Data Gather and Clean data

Schedule: Year ThreeSchedule: Year Three

Summative Analysis of program impact and evaluation of transfer to state

FOR MORE INFO...




Evaluation of training model and transferEvaluation of training model and transfer


Analyze 2005-2006 field dataAnalyze 2005-2006 field data

Mentor trainees in the field Mentor trainees in the field

Receive training Receive training

Train Virtual School Train Virtual School

Collect field Data Collect field Data Gather and Clean data

Schedule: Year FourSchedule: Year Four

Build capacity at the state level for future evaluation efforts

FOR MORE INFO...





Disseminate model to other states Disseminate model to other states Train additional WV agencies Train additional WV agencies

Collect field data on new programs Collect field data on new programs

Related LinksRelated Links

Project Calendarhttp://wvde.state.wv.us/calendarRockman et al websitehttp://rockman.com/ Virtual School websitehttp://virtualschool.k12.wv.us/vschool/ WVEIS websitehttp://wveis.k12.wv.us/

Documents

ED PACE Educational Development for Planning and Conducting Evaluations West Virginia Department of Education