Virtual School Symposium 2008 Evaluation in K-12 Online Learning Lessons from the Field

Virtual School Symposium 2008

Evaluation in K-12 Online LearningLessons from the Field

Evaluation in K-12 Online Learning

Presenters Cathy Cavanaugh , University of

Florida Martha Donaldson, Alabama

ACCESS Mickey Revenaugh, Connections

Academy Donna Scribner, VHS Inc.

Moderator: Tom Clark, TA Consulting

Special Thanks to Brian Lekander,OII


AgendaIntroductionsPublication Overviews: OII Evaluating Online Learning Guide NACOL Issue Brief (in process)

Lessons from the Field: Preparing to EvaluateConducting Effective EvaluationsUsing Evaluation Results

Q & A


Office of Innovation and ImprovementU.S. Department of Education

.

Evaluating Online

Learning: Challenges

and Strategies for

Success

The latest guide in the Innovations in Education series

Office of Innovation & Improvement, U. S. Department of

Education

Evaluation in K-12 Online LearningWhy Did OII Prepare This Guide? Continued skepticism in some

quarters about quality of online learning

Desire for greater accountability in Gov’t funding programs

Concern that innovation often outpaces what we know about educational technology

Evaluators of federally-funded projects are sometimes unfamiliar with online technologies

Evaluation in K-12 Online LearningGuide Based on Seven Case Studies Alabama ACCESS Algebra I Online (Louisiana) Appleton eSchool (Wisconsin) Arizona Virtual Academy Chicago Public Schools Virtual

High School Digital Learning Commons

(Washington) Maryland Public Television’s

Thinkport


Who is this Guide For? Online program administrators

who need to think strategically about evaluation – and how they will use it as their program evolves

Evaluators who are not very experienced with online learning and the challenges it presents

Evaluation in K-12 Online LearningWhat is the Guide’s Focus? Evaluation Challenges Commonly

Encountered in Online Evaluations

Instructive Examples of Responses to these Challenges


Six Evaluation Challenges Featured Meeting the Needs of Multiple

Stakeholders Building on the Existing Base of

Knowledge Evaluating Multifaceted Online

Resources Finding Appropriate Comparison Groups Interpreting the Impact of Program

Maturity Translating Evaluation Findings into

Action

Evaluation in K-12 Online LearningSome Things OII Has Learned Evaluation of online learning is

inherently political Programs will always be pressed for

information/results before they would like to be

It’s best to be proactive by anticipating the needs of your stakeholders

You won’t always be able to control the spin on your results

It’s helpful to think of evaluation as an ongoing process


Where Can I Get the OII Guide?

To view online: http://www.ed.gov/about/pubs/intro/innovations.html

From Ed Pubs:Order online at http://www.edpubs.gov or call 1-877-433-7827 (request order number ED004344P)


Office of Innovation and ImprovementU.S. Department of Education

.

Quality and Effectiveness in Online Learning

Issues Brief

Research Committee

North American Council for Online

Learning

Evaluation in K-12 Online LearningNACOL Research Committee Quality and Effectiveness in K –

12 Online Teaching Research Based Practices Practices currently adopted by

Online course providersPractices and Policies for K-12

Online Teaching and LearningOnline Professional Development

Standards Across North AmericaFeatures of Teaching in Virtual

Schools

Evaluation in K-12 Online LearningNACOL Research Committee Online Teacher Support Programs:

Mentoring and Coaching Models Description of the Mentoring/Coaching

relationships in vignettes from perspectives of several virtual high schools

Alabama ACCESS Distance Learning Colorado Online Learning Florida Virtual School Idaho Digital Learning Academy Mississippi Virtual School Missouri Virtual Instructional Program Tennessee: e4TN Virtual High School Global Consortium


Introduction Why evaluate online learning? To demonstrate the value or

worth of your program To improve your program over

time To document participant

outcomes related to program goals

To meet stakeholder interests/accountability needs


Introduction What happens when evaluation is

neglected? Program set up without clear goals Focus on activities and simple outputs Desired change in participant outcomes

undefined Data essential to studying success

remains undefined, ungathered Focus on anecdotal evidence,

testimonials Stakeholder information needs

neglected Program unable to demonstrate value

or worth


Preparing to Evaluate Question posed to panelists

(program managers & evaluators):Please think back to the start of the online learning program. What did you do to prepare to evaluate quality, effectiveness, and impact?

Question for current programs: How can you set up your online learning program to study quality & effectiveness over time?


Alabama ACCESS: Look at Your Specific Objectives

To Provide:

Equal Access to High Quality Instruction An Infrastructure That Delivers Quality

Learning Opportunities Greater Equity for all Alabama Public High

School Students Through Cutting-Edge Technology

Wide Range of Courses Available to Relatively few Alabama Students Today (“Advanced Diploma” Courses, Advanced Placement Courses, Additional Course Offerings, Remediation and Supplemental Resources)

Preparing to Evaluate


ACCESS—Established Need for Evaluation—Quality, Effectiveness, Impact

Built Into Program Design Purpose

Provide Evidence that Goals of Initiative are Met

Gauge Satisfaction with Courses Recommend Changes in Procedures and

Resources to Improve and Strengthen Program and Increase its Positive Impact

Determine Effectiveness of Regional Support Centers



ACCESS—Preparation for Evaluation

Development of RFP at Beginning of Initiative

Development of Evaluation Plan with ISTE Identification of:

Questions to be AnsweredData that will be NeededData that will not be AvailableEvaluation Methods to be Used

Development of Data Collection Instruments



ACCESS—What Worked

Financial support from the state. Rapid expansion of the infrastructure. Highly rated Regional Support

Centers. Thousands of students take classes

that would be otherwise unavailable. Teaching practices change with

increased technology integration and student-centered pedagogy.



ACCESS—Strategy-in-Progress

Seek out teachers with virtual and F2F students.

Compare outcomes for students in similar classes with the same teacher.



ACCESS—Evaluation Issues

Availability of data (e.g., lack of common end-of-course tests).

Access to data (e.g., Advanced Placement records held by testing vendor).

Technological barriers (e.g., firewalls prevent submission of survey responses).

Research design challenges.



VHS Global Consortium

Mission To develop and deliver standards

based student centered online courses to expand students’ educational opportunities and 21st century skills and to offer professional development to teachers to expand the scope and depth of their instructional skills.



VHS Believes that:• Student-centered online courses can

be designed and delivered to students to promote a high quality collaborative learning environment.• in which student exchange and interaction is

a valued component of the instructional process.

• Educational opportunity need not be limited by barriers of time/place/lack of qualified faculty.

• Rather, we believe that high-quality education is possible-today-for all students in all locations.

• Online education offers any school with Internet connectivity a wealth of trained, experienced faculty members qualified in numerous disciplines, for teaching a wide array of courses designed to meet the needs of all students. An innovative, standards-based curriculum delivered online offers diverse, exciting learning choices for students, and the opportunity and skills to participate in a national and global community.



• Online teaching should augment rather than replace traditional classroom teaching. • The Virtual High School's online courses are a

proven, flexible solution for schools needing an expanded curriculum, teachers seeking new horizons, parents wanting more involvement with their children's education, and a society grappling with ways to offer opportunity to all its citizens.

• The goals of education are advanced best by putting value and service first. • When schools work together in a collaborative

network such as VHS, they become part of an abundant and generous educational community that promotes the affordable sharing of professional resources

VHS Believes that:


VHS Global Consortium 1996 – Technology Innovation

Grant 5-year, $7.4 million US Department of Education

Non-profit; non-degree granting Consortium of schools

575+ member schools 30 states; 39 countries 11,000+ students



VHS Global Consortium Program Evaluation

Outsider reviewer Quality, Growth, Program Goals Surveys Superintendents, Principals,

VHS Teachers, VHS Students, VHS Site Coordinators

Published Annually Available to Public via Website

(www.goVHS.org) Quality Benchmark Indicators

(QBIs)




QBIs Growth Indicators measured

against Growth Goals Quality Indicators – tie back to

Mission & Beliefs Quality of Courses Quality of Professional Development Quality of Services & Program




Course Quality Benchmark Indicators

0%

20%

40%

60%

80%

100%

00/01 01/02 02/03 03/04 04/05 05/06 06/07

AP exam pass rate

AP exam take Rate

Course completion rate

Credit recovery rate




Course Completion Rate over Enrollments

0%10%20%30%40%50%60%70%80%90%

100%

00/01 01/02 02/03 03/04 04/05 05/06 06/07

0

2000

4000

6000

8000

10000

Course completion rate

VHS Students




PD Quality Indicators - First-year Teachers

0%

20%

40%

60%

80%

100%

02/03 03/04 04/05 05/06 06/07 07/08

PD Graduation Rate

PD EffectivenessRate




Program Services Quality Benchmark Indicators

65%

70%

75%

80%

85%

90%

95%

02/03 03/04 04/05 05/06 06/07

Membership RetentionRate

Course Seat UtilizationRate



Connections Academy Connections Academy programs

are mostly full-time and include K-8 Unique research challenges:

Seeking data from younger children and parents; no additional “program ally” such as site facilitator

Unique research benefits: Address whole learner, gather all demographics, include state test results/NCLB data



Connections Academy Built into Connections

Academy program: SIS data within our LMS Data analysis: Data views Log: Teacher communication, action Parent Satisfaction Survey StarTracker: Embedded feedback

on every lesson plus school as a whole

Measurable school and company goals



Lessons Learned/Next Steps What are some lessons learned

about effective practices in evaluating online learning?

What can we learn from the results of an evaluation?

How can we use results to improve the program? To inform stake-holders & decision-makers?


Lessons Learned/Next Steps Some Lessons Learned (OII,

2008) – Evaluations should: Effectively inform stakeholder groups Share tools and research methods Focus on outcomes, not activities Recruit willing research populations

early Obtain data access, or plan to gather it Move from formative to summative Disseminate timely information to

internal & external decision-makers


Lessons Learned/Next Steps What can we learn from an

evaluation? Satisfaction measures Quality & effectiveness measures Changes in knowledge & skills via

participation (participant outcomes) How can we use results to

improve the program? To inform stakeholders & decision-makers?


Lessons Learned/Next Steps Question posed to panelists

(program managers and evaluators): Based on your experience using or

conducting evaluations of online learning programs, please share:

1) What are some lessons you’ve learned about effective evaluation practices?

2) How has evaluation helped you improve an online learning program or demonstrate its worth?


Connections Academy When Evaluation Pays Off

Early MoVIP K-5 Results: User satisfaction high, teachers make the difference – validation of model

Mississippi K-8 Pilot: Even a small, short pilot can be positively revealing if designed with evaluation in mind

Ongoing Parent Satisfaction Surveys: Overall high rates (90%+) persist, and improvements in “iffy” areas absolutely trackable

Lessons &Next Steps


Connections Academy Lessons Learned

No substitute for familiarity: Evaluators need to be equipped to really dig into curriculum and platform

Data transmission is an art: Foster friendships between the evaluators and the program data wonks

Positive results are no guarantee: As in Mississippi example – can’t make up for lack of support

Lessons &Next Steps

Lessons &Next Steps


Connections Academy Lessons Learned

Transparency takes some getting used to: Educators are not accustomed to having practice so visible

Evaluation is only half the battle: True continuous improvement takes precision and persistence

Patience is a virtue: User satisfaction and academic results may diverge in beginning, but will converge if students stay

Lessons &Next Steps


VHS—Lessons Learned Don’t make decisions on a snap-

shot in time… use longitudinal data and look for trends

Evaluation criteria derives from Objectives; Objectives derive from Mission

It may be interesting but does it inform?

Lessons &Next Steps


VHS—Lessons Learned 2Continuous Course

ImprovementStudentProfessional Development

Need for Teacher SupportProgressElluminate SessionsFaculty Advising on graduated

scale

Lessons &Next Steps


How has the evaluation helped to improve the ACCESS program?

Year I Findings

Course Changes Needed

Implications/Improvements

Course Revisions Made Increased Use of Voice

Tools, Addition of Improved Speaking Assignments and Examples, and Use of Headphones in Foreign Language Courses

More Detailed Alignment and Gap Analysis Process

Addition of Course Development/Revision Component

Lessons &Next Steps


Year I Findings

Need for Better/Increased Communication and Interaction Among Teachers, Students, Facilitators, and Support Center Staff

Implications/Improvements Introduction of Regular

Faculty Meetings via Web Conferencing

Assignment of SDE Liaison to Each Support Center Region

Expansion of Contract for Web Conferencing Capability

Additions to Training Agenda Review of Teacher Pay Issues


Lessons &Next Steps


Year I Findings

Need for Additional Professional Development and Training Modules

Implications/Improvements Development of Additional

Training Modules Modification of

Professional Development Plan

Addition of SDE Staff Member to Coordinate Professional Development

Establishment of Teacher Mentoring Plan

Development of C.A.S.T. Site for Teachers


Lessons &Next Steps


Year I Findings Need for Assistance

With Scheduling, Registration, and Enrollment Issues

Increased Number of Students Not Prepared/Ready for Assigned Class


Development of Training Module for Counselors on the Registration Process

Expansion of Meetings with Counselors (Regional and State Meetings)

Decision to Develop a New Student Registration Site

Onsite Visits and Individualized Telephone Calls to Assist With Process

Further Look at Course Prerequisites


Lessons &Next Steps


Year I Findings

Need for Assistance With Technical Issues


Use of SDE and Regional Support Center Helpdesks

Additional School Visits Addition of Staff at SDE Equipment/Connectivity

Checks by SDE, Support Centers, and Alabama SuperComputer Authority

Increased Communication With School Staff

Identification of Key Areas of Concern


Lessons &Next Steps


Year I Findings

Need for Increased Number of Responses on Interviews and Surveys


Identification of Reasons for Poor Response Rate Timing Surveys Used Filters/Blocks at

School Level Content/Clarity

Issues


Lessons &Next Steps


Access—Recommendations for Future

• Increase Teacher Training, Including Additional Hands-on Time With ACCESS Technologies

• Further Develop Ways to Ensure High Interaction Between Teachers and Students (Such as Moving up of Timeline for Blended Model)

• Increase Role of Support Centers in Providing Training (More Targeted Training for Counselors and Facilitators)

• Continue Annual Audits of Support Centers• Continue emphasis on What Data is Available and

How it Can be Obtained• Ensure That New Student Registration & Data

System is Designed to Collect Needed Data and to Integrate Effectively with LMS for Data Collection and Reporting

Lessons &Next Steps


Q & A Time

Do you have questions for our panelists about evaluating online learning?

PPT slides will be posted at:vss2008.wikispaces.com/

field


Thank You! Thanks for your participation in

our session today!

Tom Clark (moderator)Cathy CavanaughMartha DonaldsonMickey Revenaugh

Donna ScribnerThanks to OII & NACOL

Documents

Virtual School Symposium 2008 Evaluation in K-12 Online Learning Lessons from the Field