McCoy College of Business Administration (BBA Assessment)

Why Assessment? We give grades don’t we? What is an Assessment Program? What can we do? How do we do it? Special thanks to Dr. Doug Eder and Dr. Kathryn Martell for much of

the content in this presentation.

The First and Only Goal:

To teach for long-term

retention of information

and application to new situations

--after Halpern & Hakel

Deep (expert) learning is a curriculum function.

It is not a course function.

Universities are systems

Direct vs. Indirect Assessment

Direct assessment acquires evidence about student learning and the learning environment: Exams, projects, logs, portfolios, observations....

Indirect assessment acquires evidence about how students feel about learning and their learning environment: Surveys, questionnaires, interviews, focus groups, reflective essays....

Whatever Assessment Is...

...It’s Multiple Measures Over Time.

Why Assessment Won’t Go Away

• Limits on money• Shifts in public view of Higher Education• State laws and education standards• Accountability to accrediting agencies to

maintain operating credentials

* * * * *Result: We must demonstrate quality of outcome

and return on investment.Through assessment educators meet

responsibilities to themselves, to students, and to the public.

What is “Program-Level” assessment as specified by AACSB?

The college or school must specify program-level learning goals for each separate degree program. Generally, such goals are anticipated for each degree, not for separate majors or concentrations within a degree. For example, the McCoy College offers a Bachelor of Business Administration (BBA) degree with defined majors in accounting, CIS, economics, finance, management, and marketing. A set of program-level learning goals for the BBA degree must be provided and assessed; goals for each major will not be required for AACSB program review accreditation. The MBA program has program-level learning goals reflecting graduate-level learning outcomes.

Strongest Links:The Scholarship of Teaching

Clear Goals Adequate Preparation Appropriate Methods Significant Results Effective Presentation Reflective Critique

Assessment vs. Evaluation

Assessment focuses on the student and the learning environment.

Evaluation focuses on the professor and the teaching performance.

We Give Grades, Don’t We?

Aren’t grades(by

themselves)enough?

NO!

When used alone, grades are no longer regarded as adequate indicators of student learning.

Professor autonomy: Grades in one course or section may be recorded using a different standard than grades in another.

General accountability: Much of the current public annoyance with higher education comes from a lack of skill and content mastery by students who have received good grades.

Grades may reflect many things besides student mastery of course objectives:

Verbal ability

Participation

Cooperation

Extra credit

Attendance

Effort

Criterion Performance vs. Value Added

Good Assessment Asks Good Questions

Do we have a curriculum or a set of common courses? Which of these do our students experience?

What do we want our students “to be” or “to have” when they have completed the program? What are our curricular goals?

What do our students “do” to demonstrate accomplishment of curricular goals (i.e. course objectives)? How much and what kinds of writing? What kinds of math? Oral Presentations? Analysis and Reports? Are these relevant to our curricular goals?

How do we demonstrate to ourselves that students are achieving or have achieved curricular goals?

BBA Common Core Courses

1. CIS 1323 Introduction to Microcomputer Applications. (3)

2. QMST 2333 Business Statistics (3) 3. ACC 2361 Introduction to Financial Accounting (3) 4. ACC 2362 Introduction to Managerial Accounting (3) 5. ECO 2314 Principles of Microeconomics (3) 6. ECO 2315 Principles of Macroeconomics (3) 7. BLAW 2361 Legal Environment of Business (3) 8. MGT 3353 Business Communication (3) 9. MGT 3303 Management of Organizations (3) 10. MKT 3343 Principles of Marketing (3) 11. CIS 3380 Personal Productivity and Information

Technology (3) 12. FIN 3312 Business Finance (3) 13. MGT 4335 Strategic Management and Business Policy

(3)

Bachelor of Business Administration LEARNING GOALS (BBA)Upon completion of the BBA, a student will be able to:

1. Conceptualize a complex issue into a coherent, persuasive written or oral statement.

2. Use critical thinking skills to evaluate information, solve problems, and make sound decisions.

3. Use information technology skills in decision-making.

4. Apply general concepts of ethical behavior in dealing with stakeholders.

5. Understand the importance of group dynamics in achieving organizational goals and use the skills needed for effective teamwork.

6. Understand the importance of culture and diversity.

An Assessment Question: How Do You Know...

...that students walk out your door looking like you want them to? What behaviors have they exhibited or products have they produced? What are the indicators for your goals?

Goals and Objectivesfor Students

Goals express what we want our students to be.

Objectives describe what we want our students to do.

.....

• Objectives are indicators of goals.

Faculty Teaching Common Core Courses Should:

Talk to each other (if multiple faculty teach the course) . . .

Agree on specific objectives for the course . . .

Make sure course objectives are relevant to Program-Level goals . .

To be assessment friendly, objectives should:

Focus on students Make the learning goals visible (serve

as indicators) Describe behaviors or products

(doing, making) that can be captured by assignments.

Bloom’s Taxonomy

LEVEL SOME COGNITIVE BEHAVIORS

Evaluation Appraisal of an Analysisor Synthesis

Synthesis Assembly of Application Analysis Disassembly of Application Application Use of Understanding Understanding Management of Knowledge Knowledge Memorization of facts, language,

concepts, principles, theories

The Next Step . . .

Faculty Should Agree on How Best to Measure Student Performance on each Objective Contributing to Program-Level Goals.

Some Assessment Ways and Means

Assessment days or centers Case studies Classroom assessments Completion and retention

studies Content analyses Debates Direct observations Focus groups Graduate success Internships and service

learning Interviews (including

videotapes) Exams for certification and

licensure

Matrices Performances Portfolios of several kinds Projects (Primary Trait

Analysis) Questionnaires and

surveys (Direct and telephone; employer, alumni, and student attitude and satisfaction)

Reflective essays Study and activity logs Tests

(Locally-developed and standardized)

Transcript analyses

Another Step . . .

Depending on the method (ways and means) used to assess a course objective, develop a rubric or a method of measurement.

Evaluate student performance on the specific objective on three levels: Exceeds Expectation; Meets Expectation; Does Not Meet Expectation.

Report your findings. Discuss the result(s), determine any

actions to take (if necessary).

Example Rubric for MGT. 4335Base: 100%-90% All sections were thoroughly and insightfully completed. All narratives, profiles

and relevant sections were factually accurate and documented. Lots of facts and data to support your claims. You clearly demonstrate a superior level of analytical reasoning and critical thinking based on your analysis. The paper was well-written and obviously put together with care (page breaks, layout, etc are appropriately placed, the paper is grammatically correct, and the level of writing is appropriate to an undergraduate senior in the McCoy College).

89%-80% All or most sections were adequately addressed and factually accurate. All

sections included and reasonably prepared. Most sections include data to support assertions and are appropriately documented. A large amount of analytical reasoning and critical thinking was evident. Writing and layout may contain a few errors, but most of the document is relatively clean.

79%-70% Some (few) sections were rather shallow and superficial, lacking any significant

insight, and/or a few are missing in the document. Some to little data was provided to support claims, and documentation is either incorrect or missing. Some of the sections may contain factually inaccurate data and the layout may have been sloppy and/or done without care. Writing may be fair to poor containing several errors and style is poor or inappropriate for a formal document.

69% or less The paper was extremely poor, and/or some sections were missing. The

sections were VERY superficial and little documentation was provided, and/or the sections deviate from the required format. Errors/inconsistencies are evident. Writing contains several to many errors and grammar is poor. In short, it appeared that the paper was written ‘at the last minute’ and didn’t reflect much thought.

Responsibilities of Core Course Coordinators

1. completing the Course Alignment Matrix which specifies which program-level learning goals are covered in a specific core course,

2. developing course learning objectives that lead to fulfillment of program-level learning goals,

3. collecting data from faculty teaching course sections to support or measure student learning of program-level learning goals (including a representative sample of measurement devices/techniques/assignments to be maintained in the office of the core course coordinator),

4. reporting collective results on program-level learning goals to the McCoy College Assessment Committee, and

5. discussing and acting on specific recommendations made by the college assessment and curriculum committees.

McCoy College of Business Undergraduate Learning Goals Matrix Fall 2005-Spring 2006

Course: Faculty member: Date:

Goal Covered Assessed Method of

assessment 1. Conceptualize a complex issue into a coherent, persuasive written or oral statement.

1a. Develop well-written reports, memos, and letters that explain findings, organize ideas into a coherent train of thought, and justify a conclusion or recommendation.

1b. Make effective oral presentations that explain findings, organize ideas into a coherent train of thought, and justify a conclusion or recommendation.

2. Acquire and use analytical tools and skills for evaluating information, solving problems, and making sound decisions.

2a. Synthesize and evaluate the relevance of data.

2b. Demonstrate critical and reflective thinking skills in exploring new questions, analyzing complex issues from multiple perspectives, and arriving at reasoned conclusions.

McCoy College Undergraduate (BBA) Program Goals CoverageFall 2005

Cov. Assd. Cov. Assd. Cov. Assd. Cov. Assd. Cov. Assd. Cov. Assd. Cov. Assd. Cov. Assd. Cov. Assd. Cov. Assd. Cov. Assd. Cov. Assd. Cov. Assd. Cov. Assd.1. Conceptualize a complex issue into a coherent, persuasive written or oral statement.

6 6

1a. Develop well-written reports, memos, and letters that explain findings, organize ideas into a coherent train of thought, and justify a conclusion or recommendation.

Cov. Assd. Cov. Assd. Cov. Assd. Cov. Assd. 4 4

1b. Make effective oral presentations that explain findings, organize ideas into a coherent train of thought, and justify a conclusion or recommendation.

Cov. Assd. Cov. Assd. 2 2

2. Acquire and use analytical tools and skills for evaluating information, solving problems, and making sound decisions.

17 17

2a. Synthesize and evaluate the relevance of data.

Cov. Cov. Assd. Cov. Assd. Cov. Assd. Cov. Assd. Cov. Assd. Cov. Assd. Cov. Assd. Cov. Assd. Cov. Assd. 10 9

2b. Demonstrate critical and reflective thinking skills in exploring new questions, analyzing complex issues from multiple perspectives, and arriving at reasoned conclusions.

Cov. Assd. Cov. Assd. Cov. Assd. Cov. Assd. Cov. Assd. Cov. Assd. Assd. Cov. Assd. 7 8

3. Acquire and use information technology skills in decision-making.

10 10

3a. Learn to use Information Technology (IT).

Cov. Assd. Cov. Assd. Cov. Assd. Cov. Assd. 4 4

3b. Apply IT in analysis and communication.

Cov. Assd. Cov. Assd. Cov. Assd. Cov. Assd. Cov. Assd. Cov. Assd. 6 6

4. Recognize and apply general concepts of ethical behavior in dealing with stakeholders.

7 6

4a. Respect the ethical foundations of diverse populations.

Cov. Assd. Cov. Cov. Assd. Cov. Assd. Cov. Assd. 5 4

4b. Recognize an ethical dilemma and resolve it using one of several evaluative frameworks.

Cov. Assd. Cov. Assd. 2 2

5. Understand the importance of group dynamics in achieving organizational goals and acquire and use the skills needed for effective teamwork.

5 6

5a. Understand the importance of group dynamics in achieving organizational goals.

Cov. Assd. Cov. Assd. Cov. Assd. 3 3

5b. Acquire and use the skills needed to work productively in teams.

Cov. Assd. Cov. Assd. Assd. 2 3

6. Understand the importance of culture and diversity.

9 7

6a. Comprehend the nature and significance of differences in individuals and groups based on their culture, ethnicity, religion, and other background factors.

Cov. Assd. Cov. Assd. Cov. Assd. Cov 4 3

6b. Understand the impact of globalism and multiculturalism.

Cov. Assd. Cov. Assd. Cov. Assd. Cov. Assd. Cov. 5 4

Goals Acc. 2361 Acc. 2362CIS 1323 QMST 2333 Eco 2314 Eco 2315 Blaw 2361 Mgt. 3353 CIS 3380Mgt. 3303 TotalsMgt. 4335Fin. 3312Mkt. 3343

What documentation must be retained by the Core Course Coordinators?

Core course coordinators should retain copies of instruments used for direct assessment of student learning, such as assignments, written products, test questions, rubrics, scoring grids, etc. The course coordinators also must retain sample student products relating to the instruments used. Aggregate results of direct assessments for various program-level learning goals gathered and/or submitted to the assessment committee should be retained by the course coordinator by semester or year for review. Minutes of core course faculty meetings should be retained with special emphasis given to items concerning improvements in instruments used to assess student learning and acknowledgement and review of data.

Must all students be assessed?

For the purpose of meeting AACSB-International standards relating to Assurance of Learning (16, 18 and 20), sampling may be utilized as long as it is representative. Faculty teaching college core courses for various programs, as scholars in their respective fields and under the leadership of the Core Course Coordinator, have the responsibility for determining appropriate representative sample sizes. Graduate core course coordinators may use census data from graduate core courses for assessment purposes.

Should college core course faculty and coordinators gather data and submit results every year?

Program-level learning goals for undergraduate (BBA) and graduate (MBA) programs are reviewed on a scheduled rotational basis. At least two program goals are reviewed each year, and college core courses covering that particular program goal are required to submit the results of course-embedded assessment to the assessment committee during the scheduled review year. Since the McCoy College views assessment as a continuous process, core course coordinators and core course faculty are encouraged, but not required, to gather data each semester for the reporting of assessment results.

Undergraduate Goal Rotation

Program-Level Goal Rotation Sequence: Academic Year 2005-2006: Goal 1 – Conceptualize a complex issue into a coherent,

persuasive written or oral statement. Goal 5 – Understand the importance of group dynamics in

achieving organizational goals and use the skills needed for effective teamwork.

Academic Year 2006-2007: Goal 2 – Use critical thinking skills to evaluate information,

solve problems, and make sound decisions. Goal 3 – Use information technology skills in decision-

making. Academic Year 2007-2008: Goal 4 – Apply general concepts of ethical behavior in

dealing with stakeholders. Goal 6 – Understand the importance of culture and

diversity.

We have never been asked...

...to assess everything all at once.

Who sets performance standards? How should Core Course Coordinators report student performance on various college-level learning goals?

For each program-level learning goal assessed in a college core course, the faculty will determine their minimum expectation or standard. For reporting data on any specific program-level learning goal, core course coordinators should aggregate data and report the results by three levels: Exceeds Expectations; Meets Expectations; or Does Not Meet Expectations. In order to preserve the integrity of the assessment process, core course coordinators should release aggregate data by course (all sections) and not individual results by the professor teaching the core course. Program Assessment in the McCoy College is not the evaluation of faculty performance in the classroom; it concerns the aggregate level of student learning of various program-level goals.

Enclosed in this folder are the results of Goal 1a & 1b assessment for the Fall 2005 semester. I have included a packet of evaluation sheets; each student received a sheet based on their performance on the individual case analysis. For McCoy College Learning Goal 1a, the specific items are item 8 – Bibliography & Appendices, and item 9 – Writing/Format. Greater weight was given to item 9 although item 8 was included in the assessment. The bottom of each sheet indicates whether the student exceeded expectations, met expectations, or did not meet expectations. A copy of the grading rubric is included in this packet. For Goal 1b, a single sheet included shows how each individual performed on their oral communications segment of a case analysis. Last names have been removed although first names remain for my control purposes. Fall 2005 Goal 1a Results: Exceeds Expectations: 5 students or 19.23% Meets Expectations: 18 students or 69.23% Does not meet Expectations: 3 students or 11.54% Fall 2005 Goal 1b Results: Exceeds Expectations: 5 students or 19.23% Meets Expectations: 19 students or 73.08% Does not meet Expectations: 2 students or 7.69%

Example Report on Program Goal

An Important Lesson from the Farm

A pig doesn’t get any fatter

merely by weighing it.

How many students must meet or exceed the performance standard on a college-level program goal?

There is no prescribed percentage of students that must meet the standard articulated by faculty. AACSB and other constituent groups will look at whether and how these data are used for continuous improvement purposes. Thus, a poor showing of student mastery of a given program-level learning goal would only be a concern if the curriculum was not subsequently modified to improve student skills/performance of that learning goal. A second emphasis in the Assurance of Learning Standards is that program-level learning goals are used to communicate competencies of graduates to students, employers and other constituents. Thus, faculty have the responsibility to develop and modify curricula to insure that program-level goals are achieved by most graduates and not just a portion.

What Assessment Asks of You! One day of initial development time to determine

course objectives and agreeing on a grading rubric for specific course objectives.

Two to three hours per semester (depending on the goal rotation) to extract student performance on course objectives leading to the fulfillment of program-level goals.

Writing a short report (1 page) concerning student performance.

Maintaining examples of student work, grading rubrics or other examples demonstrating student achievement of course objectives.

Talking to other faculty about continuously improving classroom processes and products to improve student learning of program level goals over time.

Closing Thought

The enemy of the good is the perfect.