Project Funded by:

E-Outcomes Assessment Project: Technology Linking Assessment & Learning

Luvai Motiwalla, Steven Tello & Kathryn Carter College of ManagementUNIVERSITY OF MASSACHUSETTS LOWELL

Presentation at

The NERCOMP ConferenceMar 20th 2007, Worcester, MA.

Project Funded by:

Agenda

Assessment Challenge Project Overview Program-Level Assessment Process Systems Integration Progress to Date Future Directions

Project Funded by:

The Assessment Challenge Increased calls for accountability in higher education

The Spelling Report, Secretary of Educationcalls for “evidence-based” assessment measures

Regional and Professional Accreditation agencies

Widespread interest in higher-educationinstitutions to define and assess student learning objectives

No resolution on how to define, measure, and achieve quality learning

Different approaches for outcomes assessment student learning within courses and academic programs student learning within specific fields or majors Pre and Post learning assessments, surveys, etc.

Project Funded by:

The Assessment Challenge

What is Assessment?Many different perspectives Administrator, Faculty, Student, Agency

“Assessment is the systematic collection, review, and use of information about educational programs undertaken for the purpose of improving student learning and development.” (Palomba & Banta, 1999)

Project Funded by:

The Assessment ChallengeProblems with Assessment

Tedious Costly Untimely Painful Useless

A labor intensive, paper-based process driven by accreditation agency visits & timelines

Project Funded by:

A Possible Solution…

Design and develop an electronic outcomes assessment process

which captures the course or student activities (e.g., assignments, discussions, or exams) within a learning management systems (LMS)

categorize it relative to the college or university’s stated learning goals or outcomes.

Project Funded by:

Project Overview

Two-year, grant-funded project UMass Lowell & Davis Educational Foundation

Use LMS to organize and report student progress toward program-level learning outcomes

Interdisciplinary focus 6 Business Core Courses, 6 General Education

Two operational foci Faculty Development Systems Development

Project Funded by:

Project Overview - Goals Improve the learning environment for

students by increasing student access to content, instructors and peers.

Improve content and assessment consistency across multiple disciplines.

Facilitate student development of academic e-portfolios.

Reduce overall administrative costs through online paperless assessment, reporting, tracking.

Project Funded by:

Assessment Process - Learning Outcomes

UniversityGeneral EducationOutcomes

COMLearning Outcomes

Breadth of Knowledge

Critical Thinking

Clear Communication

Diversity

Ethics

Self-Direction/ Collaboration

InformationLiteracy

I II III IV V VI VII VIII

Quantitative & Qualitative Skills

Decision-making&

Problem solving

Communication Skills

Diversity Impact

Ethics & Values

Team Leadership & Membership

InformationCommunication Technologies

GlobalImpact

Project Funded by:

Assessment Process – Self Reporting

Current ProcessManual, Paper-based, Self-report

Project Funded by:

GLOBAL IMPACTS

DIVERSITY IMPACT

ETHICS

IT USAGE

TEAM PARTICIPATION

QUANTITATIVE SKILLS

COMMUNICATION SKILLS

Course A

Course B

Course C

Course N

Discussion Topic A

Assignment 1

Exam 1 & 3

Assignment 2

Group Tool Usage

Group Project

Excel Assignment

Case Assignment

Assessment Process: New Approach

Proposed ProcessAutomated, Menu-driven, Drill down

Project Funded by:

Assessment Process: Future Reporting Possibilities

Percent of BSBA Curriculum

20%

15%

15%

25%

10%

10%

5%

COMMUNICATIONSKILLSQUANTITATIVE SKILLS

TEAM PARTICIPATION

INFORMATION LITERACY

ETHICS

DIVERSITY IMPACT

GLOBAL IMPACTS

0

10

20

30

40

50

60

70

CJ 44.1

45

PSY 47.10

1

ENG 42.11

0

POL 46.1

12

ECO 49.20

1

AC 60.20

1

MKT 62.21

1

OM 63.21

0

MIS 63

.310

COMMUNICATION SKILLS

QUANTITATIVE SKILLS

TEAM PARTICIPATION

INFORMATION LITERACY

ETHICS

DIVERSITY IMPACT

GLOBAL IMPACTS

Project Funded by:

Systems Integration Use LMS (WebCT Vista) as Assessment

Collection Tool Organize & present course assessment activities Present outcomes to students Collect assessment artifacts

System limitations require external database To map course activities to program-level

outcomes To organize & report progress toward outcomes

Project Funded by:

Assessment Tool

Syllabus Tool

DiscussionTool

Assignment Tool

Grade Book

Course Content Files

Groups

Chat Tool

Systems Integration

Use the activity tools in WebCT Vista Train faculty in specific use of these tools

Project Funded by:

Systems Integration

Use PowerSight to report off Grade book Train faculty to use grade book

Project Funded by:

Systems Integration

EOutcomes Database My SQL, Perl, Web Interface Export CSV from WebCT, Import to EOutcomes

Faculty map outcomes to specific course activities in EOutcomes Database Administrators then report on progress toward outcomes through EOutcomes

Project Funded by:

EOutcomes System V1

Systems Integration

Project Funded by:

Learning Outcomes Menu

Systems Integration

Project Funded by:

Progress to Date – Year 1 Conducted Summer Training Workshop Seven courses offered in Fall 2006 / Spring 2007 Developed a prototype EOutcomes database and

application

Project Funded by:

Goal 1: Improve learning environment by increasing student access…

7 Courses offered Fall2006, 385 students Each Web-Enhanced to varying degrees.

Course Course Name Enroll Instructor 42.243 Contemporary Women Writers 17 Marlowe Miller 43.308 History of Crime & Social Control 29 Joseph Lipchitz 47.101 General Psychology 35 Richard Siegel 61.301 Business Finance 155 Yash Puri 62.201 Marketing Principles 98 Srikant Vadali 66.301 Organizational Behavior TBD Stuart Freedman 63.407 Electronic Business 10 Luvai Motiwalla (PI) 66.420 Leadership Processes 41 Steven Tello (PI)

Project Funded by:

Goal 1: Improve learning environment by increasing student access…

Online Tool in WebCT No. of Courses Using Tool

Assignments 7 Course Outline 7 Syllabus 7 Gradebook 6 Lectures (slides & notes) 6 Discussion 4 Assessment 2 Quizzes, 4 Exams Groups/Teams 3

Project Funded by:

Goal 2: Improve content and assessment consistency across multiple disciplines

EOutcomes Training Institute, June 2006 (June, 2007 as well) Learning Outcomes Assessment Matrix in Excel Assessment Tools in WebCT Reporting Tools in WebCT Respondus Exam Tool EOutcomes Database Tablet PC Demo

Training Materials online http://comwebserver.uml.edu/eoutcomes/ EOutcomes WebCT Course

Project Funded by:

Goal 3: Facilitate student development of web-based academic portfolios.

ePortfolios provide ability to link sample student work to activity to learning outcomes

Very limited success due to WebCT issues and manual nature of effort

ePortfolios often focused on student need to build a portfolio, rather than administrative need to measure achievement

Continue to review possible tools within WebCT and external

Possibly EOutcomes Database

Project Funded by:

Goal 4: Reduce overall administrative costs

Possible savings include: Supply savings – less scantron sheets and paper for

exams Proctor savings – less time spent scanning, reviewing

and reporting exam grades (less room for error) Faculty time savings when:

grading exams, many are auto-scored grading quizzes, many are auto-scored filling out assessment spreadsheets collecting student work samples making copies

Savings are balanced by initial development effort on part of faculty, and maintenance

Project Funded by:

Emerging Research Questions Does the use of a LMS in a web-enhanced course

increase student awareness of program-level learning outcomes?

Do students perceive a benefit in the use of a LMS for course-level assessment activities?

Do faculty perceive a benefit in the use of a LMS for course-level assessment activities?

Does the use of a LMS facilitate the organization and reporting of program-level assessment data for accreditation purposes?

Does the LMS in a web-enhanced course have any positive impact on students’ involvement in a course or satisfaction with the course or their learning?

Project Funded by:

Next Steps

Develop EOutcomes Reporting System tools

Incorporate six more courses in 2007-2008 academic year

Conduct training institutes in Summer 2007 Measure Project Achievements Integrate Research Agenda

Project Funded by:

Thank you Prof. Luvai Motiwalla

Luvai_Motiwalla@uml.edu

Prof. Steven TelloSteven_Tello@uml.edu

EOutcomes Website

http://comwebserver.uml.edu/eoutcomes/