HERDING JAVELINAS : CoursEval Implementation at a Large Multi-Campus University

HERDING JAVELINAS :CoursEval Implementation at a Large Multi-Campus University

INTRODUCTION

Arizona State University, with more than 70,000 students in 15 colleges, has adopted CoursEval as our sole university-supported

course evaluation system.

Over the course of a single academic year, we expanded the use of CoursEval from two

colleges to nearly all online and on-ground courses.

INTRODUCTION

We will describe how we: achieved buy-in from administration,

faculty, staff, and students; retired our legacy paper and online

evaluation systems; balanced decentralized college-level

ownership with centralized facilitation, and saw considerable cost-savings, with no major problems. 

HERDING JAVELINASPaper evaluations were used for in-person classes for most of ASU

Strengths Units had control over survey design and content Familiar system, little training involved

Weaknesses Required considerable consumption of paper,

printing, and personnel time

o ASU commitment to reduce our carbon footprint Less than ideal for data retention, historical

tracking, etc.

HERDING JAVELINAS

Online Evaluation Legacy System Strengths:

Centralized administration

Reporting convenient for units

Legacy System Weaknesses

1. Used separate systems for survey administration and analysis/reporting.

2. Single survey for all units - did not meet needs of some (and did not match survey in use for on-ground sections). This was an issue for promotion/tenure.

3. No comparative/historical data options.

Legacy System Weaknesses (cont’d)

4. Access control required SQL programming by Associate Director to update database. Time consuming and could not delegate to staff.

5. Not available for on-ground courses, despite considerable demand from units.

6. Limited to single administration per traditional semester. Did not accommodate non-traditional course schedules.

7. Data analysis and reporting were time-consuming. Reports not available for several weeks after close of survey.

Our Requirements:

a. Online surveys for both online and in-person courses

b. Units can use their own survey(s)

c. Flexible administration schedules, controlled by units

d. Results available immediately

e. Release of reports (and report content) determined by units

SEARCHING FOR A SOLUTION

Our Requirements (cont’d):

f. Instructors, chairs, & dept. support staff have access to results

g. Reports available online for both current & future access/review

h. Flexible reporting format

i. Support comparisons across instructors & across time

j. Support co-taught sections

SEARCHING FOR A SOLUTION

We explored these options :

a. Re-design homegrown legacy system

b. Use one of the homegrown systems in use by ASU’s Engineering or Business schools

c. Identify a commercial system

10

DECISION:PURCHASE COMMERCIAL SYSTEM

Engineering & Business systems could not be easily modified

Re-design of legacy system would require considerable work by developers

Mary Lou Fulton Teachers College used CoursEval for years with good results. They did demo for us, and we were impressed with features

PILOT TEST

Attended Sept. 2010 Courseval User Group meeting to determine whether CoursEval would meet ASU’s needs

Worked with Brian Hopewell during the meeting to negotiate a price for a limited pilot

Fall 2010 Used CoursEval for online courses in all colleges

except Engineering & Business; included in-person courses for Technology

Participation was voluntary, but many used it

Spring 2011

CoursEval available for all online and in-person sections

Participation is still voluntary, but most made transition

TRANSITION

SPRING 2010

Scanned

119,038

Surveys

SPRING 2011

Scanned

49,336

Surveys

TRAININGFall 2010

a. Trained one user in each college to create survey items & responses, and assemble surveys. After surveys closed, trained them to design and release reports.

b. Training scheduled in one on one sessions - via WebEx, or in-person.

c. Coached users through cleaning course/instructor data from PeopleSoft before our staff imported into CoursEval

d. I handled entire administration: e-mails, reminders, & questions from faculty & students

DISTRIBUTING RESPONSIBILITIES

Winter 2010

I transferred responsibility for e-mails to college users – sending them detailed instructions (including screenshots)

I continued to manage PeopleSoft queries and data imports

Spring 2011 I was instructed to fully transfer

responsibility to the units

Several users realized that there was too much work for one person, and wanted me to train others in their areas

DISTRIBUTING RESPONSIBILITIES

SPRING 2011 CoursEval agreed to send Michele to ASU for a two-day Train the Trainer workshop. Approximately 40 users from across the university registered for the workshop.

Michele trained them on:

Creating items & responses

Building surveys

Data imports

Survey administration

Reports

SPRING TRAINING - 2011

We also taught the group how to run the PeopleSoft query for course, instructor, and student data, how to clean the data, and how to do imports.

We obtained approval for me to send a common e-mail invitation to students to reduce the number of student e-mails.

Established the ASU CoursEval User Group, so that users from across the university can support one another.

Invitations

Responses

Rate

Spring 2010

29,998 14,041 47%

Spring 2011

140,543 75,701 54%

SPRING 2011

We thought we had our pigs in a row! But then…

SPRING 2011

Shortly after the spring training, we lost several of our power users!

SUMMER 2011

More Challenges Some trained users left on vacation

during open surveys

More staff turnover

Proposed Solutions Train more people?

Return to centralized model?

FALL 2011

Stay Tuned …We are in discussions with ASU leadership about

how to balance unit-level autonomy with centralized support and continuity.

Wanda K. BakerAssociate Director for Assessment & Information TechnologyUniversity Office of Evaluation & Educational EffectivenessArizona State UniversityTempe, AZ480-965-0674wanda.baker@asu.edu