Outcomes Assessment in the Distance-Learning Environment

Outcomes Assessment in the Distance-Learning Environment

We are:

Citrus College, Glendora, California (LA County)

Cedar Valley College, Lancaster, TX (Dallas)

• Marcy Morris: Learning Center, English (CC)• Theresa Villeneuve: Communications, Fine Arts• Tom Eiland: English (CC)• Lisa Nightingale: Instructional Support & Dist. Ed,

Psychology (CVC)• Eric Nightingale: Psychology, History (CVC)

Citrus College, Glendora, California (LA County)• 11,000 Students• 100+ DE sections this

semester• As many as 4000+

students per semester

Cedar Valley College (Dallas CCCD)• 4800 students• 160+ DE sections this

semester• As many as 4000+

students per semester

• Do differences exist in outcomes assessment for the distance education / online environment vs. the on-ground, traditional environment?

• Should they be different for the DE environment?

• Citrus’ DE program & the DCCCD have been evaluating/reevaluating outcomes for 11 years.

The Questions…

DE Curriculum Outlines—The History…

The first big questions…

• 1996—discussion about whether DE courses should have a separate course outline

• contact hours—connection with students?

More History…• 1998—incorporated the Distance Education

course outline into the conventional course outline.

• Independent/Directed Study added as method of instruction.

• Instructor syllabus included which specifies how contact occurs.

The History, continued…

• March 10, 1998—California Board of Governors approved recommendations to Title 5 Regulations on Distance Education.

• Each DE course to be separately reviewed and approved.

• Emphasis on

regular effective contact

And Yet… Even More History…

• 2004: Distance Learning Course Outline Addendum

• Benefit of providing course in DE format• Methods of contact• Course objectives / outcomes /

assignments / assessments• Methods of evaluation for online mode.

Accreditation Standards...a tilt…?

• An effective institution maintains an ongoing, self-reflective dialogue about its quality and improvement (Introduction to the Accreditation Standards 2002)

But, That Was Not Enough...

• Distance Learning Manual

• published August 2006 by Accrediting Commission for Community and Junior Colleges (ACCJC) / Western Association of Schools and Colleges (WASC)

ACCJC / WASC Distance Learning Manual

• Institution: Academic standards for all courses and programs should be the same as for all other educational experiences delivered by the institution.

• Students: Students should expect that distance learning programs will permit completion of learning outcomes and objectives in the same manner as those delivered in traditional programs.

• which validated our DE curriculum “journey”

Interesting Acknowledgement: methodology used in DE classes have been incorporated into classrooms on campus• Communication:

• Email• Web• Discussion / message / announcement Board• Chat

• Other:• Websites/FAQ’s • Course and Study Materials• Turnitin.com • Sample material

The Charge

• To identify elements of quality online courses rather than elements of good instruction.

• To identify class management standards and responsibilities for teaching online.

The Action

Review of existing course standards documents/models

Quality Matters: Inter-Institutional Quality Assurance in Online Learning

A Grant Initiative of MarylandOnline

Sponsored by the U.S. Dept. Education Fund for the Improvement of Postsecondary Education (FIPSE)

FIPSE Interested Because …

• Quality assurance of courses is important

• Voluntary inter-institutional assurance has never been done before

• This can serve as a national model

Quality Matters!

For Our Purposes, Quality Is…

• More than average; more than “good enough”

• An attempt to capture what’s expected in an effective online course

• Based on research and widely accepted standards

Higher Standar

d

100

50

0

“They” say there is . . . .“No Significant Difference”

• In student outcomes between alternate modes of educational delivery . . . but

• The No Significant Difference Phenomenon (Thomas L. Russell 2001, IDECC)

• includes 355 research reports, summaries, and papers that document no significant differences (NSD) http://nosignificantdifference.wcet.info

there is a Significant Difference

• When comparing face-to-face education with Internet-Based Distance Education, there is a significant difference in favor of the Internet-Based Distance Education

. . . . In regards to student outcomes

Significant Difference - Better Results with Technology 2006 - Sahin, C. S.Overcoming the "No Significant Difference" Phenomenon in Distance Education by Internet Andolu University, Turkey http://aof.edu.tr/iodl2006"...

Example: Freshman Composition 101:

• 1996 : Created freshman reading and composition course for distance learning

• Faculty demands:• Maintain course standards

• Administration requirements:• Cannot change course outline• Only add Independent Study

Traditional Primary Learning Objectives

• Analyze Fiction Objectively • Present Research Paper• Demonstrate Use of Literary Terms in Deductive Essay • Utilize MLA Formatting

Distance Education Objectives

• Maintaining the Flexibility of Distance Education

• Providing Students with Quality Educational Experience

• Keeping Distance Ed Classes Legitimate in the Eyes of Academia

Traditional Assessment Tools

• Timed, Proctored, Deductive Essay• Analytical Research Paper• In-class Lecture• In-class Discussion• Topical Presentation

Primary Hurdle

• Primary concern among English faculty when presented with possibility of distance education Freshman Comp online course was synchronous class contact.

• Theory was that in order to understand the application of literary terms in the analysis of fiction, one must be able to practice through class conversation

Other factors . . . .• Ensuring rigidity of testing

• Time• Security• Content• Format

• Maintaining quality and quantity of written research essays as required by established college standards and IGETC agreement (Intersegmental General Education Transfer Curriculum)

Solutions…

• Classroom time / Synchronous Contact• Replaced with weekly chat room meetings• Synchronous conversation with students• Chat room transcript archived on web site

for review and retrieval• All students required to attend or to read

archive and respond

Solutions…continued

• Testing Protocol:• Establishment of Proctored Testing Center• Testing center controls:

• Allowed materials• Allotted time• Identification of test-taker• Adapted Testing needs

More Solutions

• Term Papers:• Strict adherence to established guidelines

• Content• Length• Format

• Peer editing asynchronously through message board

• Turnitin.com• SafeAssign

Solutions… (almost done)

• Asynchronous Contact• Message board• E-mail• Web page• Archived chat room• Web links

Results

• Comparing ENGL 101 and ENGL 103 Freshman Composition Courses

• Retention is higher in on-campus courses• Traditional Avg: 22.5/30• Distance Learning Avg: 15.3/30

Results…continued

• Grade range reflects a standard curve in traditional courses (frown)

• Traditional Delivery Averages:• A: 18%• B: 36%• C: 24%• D: 22%

More Results…

• Distance education grade range is wider (smile)

• Distance Education Grade Averages:• A: 38%• B: 23%• C: 8%• D: 31%

FACTORS (as determined by observation, surveys and interviews)

• Population• Age (older is better)• Experience (older is still better)• Distractions (less is better)• Motivation (more is better)

• Delivery Method• Technology (BB, attachments, chat room)• Influence of real, live person

What It Means . . . .

• Delivery methods create some different outcomes BUT…

• Assessments tools and criteria are the same• Testing center more uniform than individual

classroom practices (now being used for traditional courses as well)

• Technical Support would help many students succeeded in distance education

• The distance education version of freshman composition at Citrus College has as much, if not more, student learning outcome assessment and verification than the on-campus version of the same course.

Art 199

Motion Picture Appreciation

Distance Ed vs. Traditional Deliveries

Content Of The Course

• Social/technical factors of film production

• Genres• Aesthetics• Narrative and non-narrative

construction• Film form and history• Ability to make sophisticated small talk

at cocktail parties.

Concerns

• Logistical concerns re: films• Rigor• Quality and Quantity of education to

maintain IGETC agreement

• I am the only faculty member who teaches this course.

Methods of Instruction(from course outline)

• Lecture• Lecture/Discussion• Independent Study• Out of class film viewing• Directed studyDirected study

Assessment

• Discussion• Large Paper• Quizzes

• Taken online

• Tests• Taken in testing center under controlled

conditions until this semester

What it means• No one flies under the radar • No one can sleep through a film and

pass my class• Testing is controlled• Criteria are unchanged• At least the same learning outcomes

as a traditional course.

0

2

4

6

8

10

12

14

16

As Bs Cs Ds Fs

Distance Ed

Traditional

Overall Midterm Comparison, Fall 2006

Final Comparison, Fall 2006

0

2

4

6

8

10

12

14

As Bs Cs Ds Fs

Distance Ed

Traditional

CVC Online Success & RetentionOnline Course Success and Retention

0%

20%

40%

60%

80%

100%

Spring 2005 Fall 2005

Semester

%

Overall Success

Overall Retention

Spring 2005 Fall 2005

Overall Success 58% 56%

Overall Retention 81% 79%

Enrollments 1939 2066

CVC Spring 2006 Distance Learning and On Campus Grade Distributions

Course Instr. Method College Success Retention

ACCT-2301 Prin Accounting I   On Campus 47% 70%

ACCT-2301-3427 INET Distance Learning 50% 75%

BIOL-1408 Biology Non-Sci Maj I   On Campus 58% 79%

BIOL-1408-3460 INET Distance Learning 17% 50%

BIOL-1408-3461 INET Distance Learning 53% 89%

BIOL-1408-3462 INET Distance Learning 58% 75%

BIOL-1408-3463 INET Distance Learning 49% 78%

BIOL-1408-3464 INET Distance Learning 80% 100%

BIOL-1409 Biology Non-Sci Maj II   On Campus 70% 84%

BIOL-1409-3460 INET Distance Learning 71% 100%

BIOL-1409-3461 INET Distance Learning 56% 78%

BIOL-1409-3462 INET Distance Learning 54% 85%

BMGT-1303 Prin of Management   On Campus 60% 81%

BMGT-1303-3426 INET Distance Learning 100% 100%

BMGT-1303-3427 INET Distance Learning 42% 82%

BUSI-1301 Intro to Business   On Campus 54% 84%

BUSI-1301-3426 INET Distance Learning 100% 100%

BUSI-1301-3427 INET Distance Learning 50% 78%

COSC-1300 Computer Literacy   On Campus 64% 82%

COSC-1300-3426 INET Distance Learning 67% 93%

COSC-1300-3428 INET Distance Learning 60% 84%

COSC-1401 Microcomp. Conc./Apps.   On Campus 77% 89%

COSC-1401-3426 INET Distance Learning 82% 88%

COSC-1401-3427 INET Distance Learning 44% 80%

COSC-1401-3428 INET Distance Learning 39% 78%

COSC-1401-3429 INET Distance Learning 100% 100%

CVC Spring 2006 Distance Learning and On Campus Grade Distributions

ECON-2301 Prin Economics   On Campus 63% 80%

ECON-2301-3426 INET Distance Learning 33% 33%

ECON-2301-3427 INET Distance Learning 49% 72%

ECON-2302 Prin Economics   On Campus 66% 78%

ECON-2302-3426 INET Distance Learning 61% 61%

ENGL-1301 Composition I   On Campus 58% 78%

ENGL-1301-3420 INET Distance Learning 45% 55%

ENGL-1301-3421 INET Distance Learning 50% 91%

HIST-1301 History U S   On Campus 55% 80%

HIST-1301-3420 INET Distance Learning 63% 73%

HIST-1301-3421 INET Distance Learning 57% 57%

HIST-1302 History U S   On Campus 70% 85%

HIST-1302-3420 INET Distance Learning 59% 84%

NUTR-1322 Prin of Nutrition   On Campus 71% 81%

NUTR-1322-3426 INET Distance Learning 36% 62%

PSYC-2301 Intro Psychology   On Campus 62% 83%

PSYC-2301-3420 INET Distance Learning 56% 80%

SOCI-1301 Intro Sociology   On Campus 68% 84%

SOCI-1301-3420 INET Distance Learning 81% 95%

SOCI-1301-3421 INET Distance Learning 63% 92%

SPCH-1311 Intro Speech Comm   On Campus 71% 85%

SPCH-1311-3420 INET Distance Learning 73% 87%

What Makes DE Work? . . . INFRASTRUCURE!

A viable Distance Learning program requires changes to the infrastructure of campus to sustain learning outcomes mandated by:

• College District• State Regulations• Accrediting Commission• Federal Government—Title V

Infrastructure ChangesTesting Center• Grew out of need to offer timed, proctored

testing for DE students to establish controls for test taking to insure quality

• Meets regular effective contact requirements to authenticate outcomes

• Now serves DE as well as the larger campus community--academic makeup testing, assessment testing, adapted testing accommodation, job applicant testing, computer skills competency testing.

Infrastructure—Supporting Services of SLO’s in DE

• Distance Education Department• Administrative, Clerical, and Technology

Support Staff• Training for Students and New DE Instructors• Orientation of New Students and New Faculty• Online Services including registration and

tutoring• DE Student Support staff (between technical &

counseling)

The End?

• DE was implemented long before and is keeping pace with if not exceeding SLO requirements for traditional courses

• Distance Education has clearly prompted the college to reevaluate outcomes, has affected how we teach in the traditional classroom, and has effected a change in infrastructure to meet needs in regards to outcomes.