Embed Size (px)
Citation preview
Outcomes Assessment in the Distance-Learning Environment
We are:
Citrus College, Glendora, California (LA County)
Cedar Valley College, Lancaster, TX (Dallas)
• Marcy Morris: Learning Center, English (CC)• Theresa Villeneuve: Communications, Fine Arts• Tom Eiland: English (CC)• Lisa Nightingale: Instructional Support & Dist. Ed,
Psychology (CVC)• Eric Nightingale: Psychology, History (CVC)
Citrus College, Glendora, California (LA County)• 11,000 Students• 100+ DE sections this
semester• As many as 4000+
students per semester
Cedar Valley College (Dallas CCCD)• 4800 students• 160+ DE sections this
semester• As many as 4000+
students per semester
• Do differences exist in outcomes assessment for the distance education / online environment vs. the on-ground, traditional environment?
• Should they be different for the DE environment?
• Citrus’ DE program & the DCCCD have been evaluating/reevaluating outcomes for 11 years.
The Questions…
DE Curriculum Outlines—The History…
The first big questions…
• 1996—discussion about whether DE courses should have a separate course outline
• contact hours—connection with students?
More History…• 1998—incorporated the Distance Education
course outline into the conventional course outline.
• Independent/Directed Study added as method of instruction.
• Instructor syllabus included which specifies how contact occurs.
The History, continued…
• March 10, 1998—California Board of Governors approved recommendations to Title 5 Regulations on Distance Education.
• Each DE course to be separately reviewed and approved.
• Emphasis on
regular effective contact
And Yet… Even More History…
• 2004: Distance Learning Course Outline Addendum
• Benefit of providing course in DE format• Methods of contact• Course objectives / outcomes /
assignments / assessments• Methods of evaluation for online mode.
Accreditation Standards...a tilt…?
• An effective institution maintains an ongoing, self-reflective dialogue about its quality and improvement (Introduction to the Accreditation Standards 2002)
But, That Was Not Enough...
• Distance Learning Manual
• published August 2006 by Accrediting Commission for Community and Junior Colleges (ACCJC) / Western Association of Schools and Colleges (WASC)
ACCJC / WASC Distance Learning Manual
• Institution: Academic standards for all courses and programs should be the same as for all other educational experiences delivered by the institution.
• Students: Students should expect that distance learning programs will permit completion of learning outcomes and objectives in the same manner as those delivered in traditional programs.
• which validated our DE curriculum “journey”
Interesting Acknowledgement: methodology used in DE classes have been incorporated into classrooms on campus• Communication:
• Email• Web• Discussion / message / announcement Board• Chat
• Other:• Websites/FAQ’s • Course and Study Materials• Turnitin.com • Sample material
The Charge
• To identify elements of quality online courses rather than elements of good instruction.
• To identify class management standards and responsibilities for teaching online.
The Action
Review of existing course standards documents/models
Quality Matters: Inter-Institutional Quality Assurance in Online Learning
A Grant Initiative of MarylandOnline
Sponsored by the U.S. Dept. Education Fund for the Improvement of Postsecondary Education (FIPSE)
FIPSE Interested Because …
• Quality assurance of courses is important
• Voluntary inter-institutional assurance has never been done before
• This can serve as a national model
Quality Matters!
For Our Purposes, Quality Is…
• More than average; more than “good enough”
• An attempt to capture what’s expected in an effective online course
• Based on research and widely accepted standards
Higher Standar
d
100
50
0
“They” say there is . . . .“No Significant Difference”
• In student outcomes between alternate modes of educational delivery . . . but
• The No Significant Difference Phenomenon (Thomas L. Russell 2001, IDECC)
• includes 355 research reports, summaries, and papers that document no significant differences (NSD) http://nosignificantdifference.wcet.info
there is a Significant Difference
• When comparing face-to-face education with Internet-Based Distance Education, there is a significant difference in favor of the Internet-Based Distance Education
. . . . In regards to student outcomes
Significant Difference - Better Results with Technology 2006 - Sahin, C. S.Overcoming the "No Significant Difference" Phenomenon in Distance Education by Internet Andolu University, Turkey http://aof.edu.tr/iodl2006"...
Example: Freshman Composition 101:
• 1996 : Created freshman reading and composition course for distance learning
• Faculty demands:• Maintain course standards
• Administration requirements:• Cannot change course outline• Only add Independent Study
Traditional Primary Learning Objectives
• Analyze Fiction Objectively • Present Research Paper• Demonstrate Use of Literary Terms in Deductive Essay • Utilize MLA Formatting
Distance Education Objectives
• Maintaining the Flexibility of Distance Education
• Providing Students with Quality Educational Experience
• Keeping Distance Ed Classes Legitimate in the Eyes of Academia
Traditional Assessment Tools
• Timed, Proctored, Deductive Essay• Analytical Research Paper• In-class Lecture• In-class Discussion• Topical Presentation
Primary Hurdle
• Primary concern among English faculty when presented with possibility of distance education Freshman Comp online course was synchronous class contact.
• Theory was that in order to understand the application of literary terms in the analysis of fiction, one must be able to practice through class conversation
Other factors . . . .• Ensuring rigidity of testing
• Time• Security• Content• Format
• Maintaining quality and quantity of written research essays as required by established college standards and IGETC agreement (Intersegmental General Education Transfer Curriculum)
Solutions…
• Classroom time / Synchronous Contact• Replaced with weekly chat room meetings• Synchronous conversation with students• Chat room transcript archived on web site
for review and retrieval• All students required to attend or to read
archive and respond
Solutions…continued
• Testing Protocol:• Establishment of Proctored Testing Center• Testing center controls:
• Allowed materials• Allotted time• Identification of test-taker• Adapted Testing needs
More Solutions
• Term Papers:• Strict adherence to established guidelines
• Content• Length• Format
• Peer editing asynchronously through message board
• Turnitin.com• SafeAssign
Solutions… (almost done)
• Asynchronous Contact• Message board• E-mail• Web page• Archived chat room• Web links
Results
• Comparing ENGL 101 and ENGL 103 Freshman Composition Courses
• Retention is higher in on-campus courses• Traditional Avg: 22.5/30• Distance Learning Avg: 15.3/30
Results…continued
• Grade range reflects a standard curve in traditional courses (frown)
• Traditional Delivery Averages:• A: 18%• B: 36%• C: 24%• D: 22%
More Results…
• Distance education grade range is wider (smile)
• Distance Education Grade Averages:• A: 38%• B: 23%• C: 8%• D: 31%
FACTORS (as determined by observation, surveys and interviews)
• Population• Age (older is better)• Experience (older is still better)• Distractions (less is better)• Motivation (more is better)
• Delivery Method• Technology (BB, attachments, chat room)• Influence of real, live person
What It Means . . . .
• Delivery methods create some different outcomes BUT…
• Assessments tools and criteria are the same• Testing center more uniform than individual
classroom practices (now being used for traditional courses as well)
• Technical Support would help many students succeeded in distance education
• The distance education version of freshman composition at Citrus College has as much, if not more, student learning outcome assessment and verification than the on-campus version of the same course.
Art 199
Motion Picture Appreciation
Distance Ed vs. Traditional Deliveries
Content Of The Course
• Social/technical factors of film production
• Genres• Aesthetics• Narrative and non-narrative
construction• Film form and history• Ability to make sophisticated small talk
at cocktail parties.
Concerns
• Logistical concerns re: films• Rigor• Quality and Quantity of education to
maintain IGETC agreement
• I am the only faculty member who teaches this course.
Methods of Instruction(from course outline)
• Lecture• Lecture/Discussion• Independent Study• Out of class film viewing• Directed studyDirected study
Assessment
• Discussion• Large Paper• Quizzes
• Taken online
• Tests• Taken in testing center under controlled
conditions until this semester
What it means• No one flies under the radar • No one can sleep through a film and
pass my class• Testing is controlled• Criteria are unchanged• At least the same learning outcomes
as a traditional course.
0
2
4
6
8
10
12
14
16
As Bs Cs Ds Fs
Distance Ed
Traditional
Overall Midterm Comparison, Fall 2006
Final Comparison, Fall 2006
0
2
4
6
8
10
12
14
As Bs Cs Ds Fs
Distance Ed
Traditional
CVC Online Success & RetentionOnline Course Success and Retention
0%
20%
40%
60%
80%
100%
Spring 2005 Fall 2005
Semester
%
Overall Success
Overall Retention
Spring 2005 Fall 2005
Overall Success 58% 56%
Overall Retention 81% 79%
Enrollments 1939 2066
CVC Spring 2006 Distance Learning and On Campus Grade Distributions
Course Instr. Method College Success Retention
ACCT-2301 Prin Accounting I On Campus 47% 70%
ACCT-2301-3427 INET Distance Learning 50% 75%
BIOL-1408 Biology Non-Sci Maj I On Campus 58% 79%
BIOL-1408-3460 INET Distance Learning 17% 50%
BIOL-1408-3461 INET Distance Learning 53% 89%
BIOL-1408-3462 INET Distance Learning 58% 75%
BIOL-1408-3463 INET Distance Learning 49% 78%
BIOL-1408-3464 INET Distance Learning 80% 100%
BIOL-1409 Biology Non-Sci Maj II On Campus 70% 84%
BIOL-1409-3460 INET Distance Learning 71% 100%
BIOL-1409-3461 INET Distance Learning 56% 78%
BIOL-1409-3462 INET Distance Learning 54% 85%
BMGT-1303 Prin of Management On Campus 60% 81%
BMGT-1303-3426 INET Distance Learning 100% 100%
BMGT-1303-3427 INET Distance Learning 42% 82%
BUSI-1301 Intro to Business On Campus 54% 84%
BUSI-1301-3426 INET Distance Learning 100% 100%
BUSI-1301-3427 INET Distance Learning 50% 78%
COSC-1300 Computer Literacy On Campus 64% 82%
COSC-1300-3426 INET Distance Learning 67% 93%
COSC-1300-3428 INET Distance Learning 60% 84%
COSC-1401 Microcomp. Conc./Apps. On Campus 77% 89%
COSC-1401-3426 INET Distance Learning 82% 88%
COSC-1401-3427 INET Distance Learning 44% 80%
COSC-1401-3428 INET Distance Learning 39% 78%
COSC-1401-3429 INET Distance Learning 100% 100%
CVC Spring 2006 Distance Learning and On Campus Grade Distributions
ECON-2301 Prin Economics On Campus 63% 80%
ECON-2301-3426 INET Distance Learning 33% 33%
ECON-2301-3427 INET Distance Learning 49% 72%
ECON-2302 Prin Economics On Campus 66% 78%
ECON-2302-3426 INET Distance Learning 61% 61%
ENGL-1301 Composition I On Campus 58% 78%
ENGL-1301-3420 INET Distance Learning 45% 55%
ENGL-1301-3421 INET Distance Learning 50% 91%
HIST-1301 History U S On Campus 55% 80%
HIST-1301-3420 INET Distance Learning 63% 73%
HIST-1301-3421 INET Distance Learning 57% 57%
HIST-1302 History U S On Campus 70% 85%
HIST-1302-3420 INET Distance Learning 59% 84%
NUTR-1322 Prin of Nutrition On Campus 71% 81%
NUTR-1322-3426 INET Distance Learning 36% 62%
PSYC-2301 Intro Psychology On Campus 62% 83%
PSYC-2301-3420 INET Distance Learning 56% 80%
SOCI-1301 Intro Sociology On Campus 68% 84%
SOCI-1301-3420 INET Distance Learning 81% 95%
SOCI-1301-3421 INET Distance Learning 63% 92%
SPCH-1311 Intro Speech Comm On Campus 71% 85%
SPCH-1311-3420 INET Distance Learning 73% 87%
What Makes DE Work? . . . INFRASTRUCURE!
A viable Distance Learning program requires changes to the infrastructure of campus to sustain learning outcomes mandated by:
• College District• State Regulations• Accrediting Commission• Federal Government—Title V
Infrastructure ChangesTesting Center• Grew out of need to offer timed, proctored
testing for DE students to establish controls for test taking to insure quality
• Meets regular effective contact requirements to authenticate outcomes
• Now serves DE as well as the larger campus community--academic makeup testing, assessment testing, adapted testing accommodation, job applicant testing, computer skills competency testing.
Infrastructure—Supporting Services of SLO’s in DE
• Distance Education Department• Administrative, Clerical, and Technology
Support Staff• Training for Students and New DE Instructors• Orientation of New Students and New Faculty• Online Services including registration and
tutoring• DE Student Support staff (between technical &
counseling)
The End?
• DE was implemented long before and is keeping pace with if not exceeding SLO requirements for traditional courses
• Distance Education has clearly prompted the college to reevaluate outcomes, has affected how we teach in the traditional classroom, and has effected a change in infrastructure to meet needs in regards to outcomes.
