2004 Southern Business Administration Association Meeting, Myrtle Beach, South Carolina The UCF College of Business Approach to Assessing Student Learning

2004 Southern Business Administration Association Meeting, Myrtle Beach, South Carolina

The UCF College of Business Approach to Assessing Student Learning

Taylor Ellis, Ph.D. Patrick Moskal, Ph.D.Associate Dean Director, Testing & Evaluation

University of Central Florida

Presentation Outline1. UCF background.

2. Overview of the assessment process in the College of Business Administration.

3. Development of student learning outcomes.

4. Description and role of our Testing Lab in the assessment process.


UCF Background Large metropolitan university with over 44,000 students. College of Business Administration (CBA) offers 7

undergraduate, 10 Masters, and 5 Doctoral degree programs.

~140 full-time and 60 part-time faculty in CBA. ~8,000 students enrolled in CBA each year.


Issues Faced in UCF CBA Number of students. Required courses in core. Pre-requisites in majors. Computer Testing Lab.


CBA Assessment ProcessUniversityAssessment

Committee (UAC)

CBA DepartmentalReview Committee

(Dept. Chairs)

CBA Teaching &Learning Committees

(Dept. level)Director of Testing &

Evaluation

Develop learningobjectives,

outcomes, andperformance

measures

Supports T & L committees develop SLO’s

Associate Dean ofUndergraduate

Programs

CBA representative to UAC (UAC Chair)CBA’s DRC Chair


CBA Assessment Process (continued)


Director of Testing &Evaluation

Develop learningobjectives,

outcomes, andperformance

measures(department level)

Student Records(University Database)

CBA AccreditationDatabase

Data Analyses

Reports(to CBA, UCF,Accreditating

Agencies, and Stateof Florida)

Paper-basedassessments (e.g.,

written exams,papers,

presentations,portfolios,

standardized exams

Computer-basedassessment in

Testing Lab

Supports T & L committees develop SLO’s

Associate Dean ofUndergraduate

Programs

Assessment Process

CBA Departmental Review Committee (DRC) DRC consists of CBA Department Chairs. DRC Chair participates on the UAC and reports CBA

assessment results. Supported by Director of Testing and Evaluation.


Assessment Process

CBA Teaching and Learning Committees Selected for each degree program area. Establish student learning outcomes (SLOs) and ensure

data collection. Participate in an annual review of assessment results,

and plans for upcoming year.


Assessment Process Each degree program will develop 4-10 general

learning objectives (expectations). Each objective will be evaluated by multiple

outcome measures (exam questions or assessment types).

Summative scales will be developed to measure most learning outcomes.

Allows for evaluation in multiple classes. Allows for pinpointing specific weak areas.


Assessment Process

Degree Program Learning Expectations

Expectation 1 Expectation 2 Expectation 3, etc.

Class 1 Class 2 Class 3, etc.

Affective 1 Cognitive 1 Behavior 1

Questions & Beliefs 1-? Questions 1-? Questions & Projects 1-?ProjectsCollaboration score


Assessment GoalsEstablish student learning outcomes for each degree

program, based on the CBA Mission.Acquire data that provide direction for program

improvement and meet both AACSB and regional accreditation requirements.

Develop comprehensive data collection and reporting system.

Collect multiple measures over time (data triangulation).Utilize existing data where possible. Cause minimal intrusion on normal faculty workload.


Why Do We Need Student Learning Outcomes?

Provide feedback to students about their progress. Communicate instructional goals to stakeholders (e.g.,

trustees, community, university administration, faculty, accrediting agencies, parents).

Provide the means for assessing student learning.


Learning Objectives

Begin by developing a list of goals and objectives, such as the following “essential competencies” for all UCF Business graduates.

BE2010 Competencies:TeamworkCommunicationCreative ThinkingAdapting to changeDiversityEthics


Learning Objectives From our goals, we develop specific learning objectives. An objective is defined as a quantifiable and/or observable achievement accomplished under specifiable conditions.

How do we develop objectives? Start by answering basic questions like:

What do we want our students to know when they finish our course or program?

What should our students be able to do when they finish our course or program?


Objectives Vs. Outcomes In general, an objective is what we hope that we will

see, and an outcome is what we did see. Learning objectives define the outcomes of

instruction in clear terms. Outcomes must be both observable and measurable. All objectives/outcomes should include three primary

components: the action or behavior to be measured, conditions of learning, and standards of performance.


You Want SMART Objectives Specific (Clear and definable terms describing the

knowledge, skills, and attitudes)

Measurable (Reliable data that can be assessed in multiple ways)

Attainable (Feasible to be accurate; need not be perfect)

Realistic (Expected standards must be practical)

Time-bound (When will results will be observed?)


Types and levels of student learning outcomes Cognitive: intellectual skills.

Knowledge, comprehension, application, analysis, synthesis, evaluation (from Bloom, 1956).

Affective: what a student believes (values). Behavioral: psychomotor skills.


General Knowledge and Skills Examples

Communication Organizational Quantitative Critical Thinking Interpersonal Skills and Awareness Group Dynamics Motivational and Leadership Ability to Learn and Train Others Ability to Manage Information Technology Global Awareness Understanding Issues of Diversity Ethical Awareness


CBA Assessment Types Course embedded assessments of individual and group

knowledge and skills (e.g., exams, projects). Program level exams (e.g., doctoral candidacy exams). Certification and standardized exams. Direct observation. Teamwork. Student portfolios. Internships and service learning. Student records. College, AACSB, and employer surveys. Placement/employer data.


CBA AccreditationDatabase

Data Analyses

Student Records(Peoplesoft)

Exam Records(InQsit & WebCT):

selected test questions,student responses,

course and student info,applicable studentlearning outcomes

Other Assessment Records:assessment type (e.g.,

portfolio, oral presentation,written communication,collaborative project),

student score onassessment) course andstudent info, applicable

student learning outcomes

Learning OutcomeRecords:

student learning outcomes,outcome classification

(Bloom), department andcourse info

Reports

Faculty Informationcredentials, course syllabi

CBA Accreditation Database Development Plan

CBA Assessment Database Maintain results for each student. Prefer automated input from electronic testing. Allow input from additional sources. Allow data analyses. Create reports specific to each degree program

and for diverse stakeholders.


CBA Testing Laboratory

140 networked PCs Positioned to maintain exam security. Netsupport School Pro software to manage PCs

(e.g., monitor, block unapproved access). WebCT and InQsit1 packages available for exams. Online student exam scheduling.

1 Developed at Ball State University.





Over 7,500 exams given in fall 2003.

Over 12,500 exams given in spring 2004.

Goal is to automatically extract specified program evaluation and accreditation data from course exams.


Testing Software Features

InQsit: Easy to use and generate tests (Word, text files, Publisher’s test banks).

WebCT: Complete course management system.

Test Question Types: T/F, multiple choice, multiple response, short answer, matching, identification of objects on chart/photo (InQsit), limited essay grading (InQsit), ungraded essay, survey.


Testing Software Features (continued)

Random selection of questions from a question pool. Randomization of questions and answer choices. Adaptive scoring. Use of audio and video in tests. Start and stop times may be set by date and time. Analysis of Results: automatic student grading and item

analysis.


Testing Lab Components Under Development

Lab entrance will have student ID reader. Students swipe card or enter PID for computer verification.

Proctor software will allow the faculty/proctor PC to display:Student name and photo.Test the student is authorized to take.Computer assigned for exam.Seat reservation duration.


Questions?


For more information, contact:

Patrick J. Moskal, Ph.D.University of Central Florida College of Business AdministrationUndergraduate Student Services, BA-II, Room 101GOrlando, Florida [email protected]


Documents

2004 Southern Business Administration Association Meeting, Myrtle Beach, South Carolina The UCF College of Business Approach to Assessing Student Learning