1 Assurance of Learning in Business Schools: Observations, Implementation Issues & Guidance Thomas G. Calderon Chair & Professor of Accounting/ Director,

1

Assurance of Learning in Business Schools: Observations, Implementation Issues & Guidance

Thomas G. CalderonChair & Professor of Accounting/

Director, CBA Quality AssessmentCollege of Business AdministrationThe University of AkronAkron, OH 44325-4802Email: [email protected] Tel: 330 972-6228

Material for this presentation is based on my experience in leading assessment projects since 1996, my various research projects, the AAA’s Teaching & Curriculum Section’s Best Practices in Accounting Program Assessment (Calderon, Green & Harkness) and Kathryn Martell & Thomas G. Calderon, Assessment of Student Learning in Business Schools: Best Practices, Each Step of the Way. Vol. 1&2. AIR/AACSB. 2005. http://aacsb.edu/resource_centers/assessment/default.asp

APLG/FSA 2006 Annual SeminarFebruary 12–14, 2006http://aaahq.org/aplg/seminars/2006/program.htm

2

About this Presentation

This presentation covers:– Assurance of Learning Fundamentals– Six Observations on Assurance of Learning– Lessons learned from each observation

The presentation offers practical guidance for implementing assurance of learning projects and identifies key issues that need attention

3

Congruency, Alignment & Change

Learning goals for each program determined by the faculty and based on the program’s mission

Curriculum alignment Evidence showing that students are achieving

program’s learning goals Use evidence to initiate change and continuous

improvement

5

Curriculum Alignment

What are the program’s learning goals? Where in the curriculum are these goals

addressed? What is taught and how are students

assessed?

Program Mission

Program Learning

Goals

Courses/Co-Curr.

6

Curriculum Alignment – From AIS CourseLearning Goal (Students Will….) Covered? Briefly describe what you do (how covered and how

assessed)

1. Demonstrate knowledge and understanding of core accounting fundamentals (financial reporting, cost and management accounting, auditing, tax, and systems).

Yes This course (i) covers the basic AIS concepts and principles, business processes, data modeling methods, systems documentation techniques, data normalization, database approaches, internal control frameworks, fraud, auditing computer based systems, and systems analysis and design issues (ii) applies various concepts and principles to the five transaction cycles (i.e., revenue, expenditure, payroll, manufacturing and financial cycles). Special emphasis is placed on COSO, SOX and CoBIT frameworks.

3. Demonstrate the ability to think creatively and apply their knowledge of accounting fundamentals in innovative ways.

Yes Students work both individually and in groups to examine internal control issues in various accounting cycles. Further, students work in a semester long project on an emerging issue in AIS. As part of the requirements, students write a detailed report and make a formal presentation before the class.

6. Demonstrate ability to research an issue, analyze qualitative and quantitative data, and integrate information from multiple sources.

Yes The semester long project requires students to research an emerging issue, analyze various issues and prepare a formal report. Students leverage Ohiolink sources, Web based sources and library to conduct research and analysis. Another assignment requires students to analyze SOX section 404 material weaknesses in selected companies’ 10K and 10Q reports accessed from the SEC’s website.

7

Evidence

Not Course Grades Direct based on actual samples of students’ work

linked to program learning objectives Indirect attitudes, opinions and assumptions about

learning linked to program learning objectives Must be “public” shared with faculty in the program Must be based on individual student performance

unless assessing teamwork May be shared with students

8

Closing the loop

Analyze and report evidence Share evidence with faculty and stakeholders Discuss evidence and implications Use evidence in decision making processes Act on evidence, propose improvements Implement proposals and continuously

improve

9

Plan Before You Leap!

Before starting, develop an assessment plan that includes:– Learning objectives– Evidence to be used– Method to collect evidence– Desired expectations– Who is primarily responsible for collecting evidence– When and how will evidence be reported, discussed and used

Follow the plan

10

Observations and Practical Lessons

Six Observations on Assurance of Learning Lessons learned from each observation Examples and implementation ideas

11

Assessment and the apparent dominance of indirect evidence

Observation 1: – Business schools and accounting programs

use multiple types of evidence to assess learning

– Indirect evidence appears to dominate

12


Type of Evidence FrequencyPercent

(N=160) Type

Student Survey 114 71 Indirect

Alumni Survey 106 66 Indirect

Required Capstone or Senior Project 92 58 Direct

Business Advisory Board input 88 55 Indirect

Employer Survey 83 52 Indirect

Syllabi Review 77 48 Indirect

ETS Major Field Test 66 41 Direct

EBI surveys 60 38 Indirect

Course embedded assessment with program standards (rubric) 59 37 Direct

Student Interviews 55 34 Indirect

Student GPAs 45 28 Indirect

Focus Groups 41 26 Indirect

13


Lesson 1: Recognize important characteristics of assessment evidence– Must be appropriate within the context of the

school’s mission– Focus is on degree program– Must relate to specific learning objectives – Place emphasis on direct evidence of student

learning– Use indirect evidence to supplement direct evidence

14

Critical pragmatism vs. scientific orthodoxy in evidence collection

Observation 2:– Measurement is a concern for everyone– There is an inherent quest to “measure” learning– Measurement comes with certain connotations

Precise Objective and Unbiased Representationally faithful Valid and reliable

– Assessment data are sometimes criticized as unreliable and unscientific, having high inter-rater variability, and difficult to develop and use

15


Lesson 2: Recognize vital evidence collection and analysis issues– Root evidence collection and measurement in

common sense and use evidence that facilitate dialogue and understanding of student learning.

– Avoid fixation on traditional validity concepts; rather employ the contemporary holistic approach to validity – validity in use.

– Be aware of possible scoring variability among different raters. How much are you willing to accept?

16


Lesson 2 Continued: Recognize vital evidence collection and analysis issues– Build consensus at alignment stage and in

planning for assessment (e.g., standardized rubric development)

– Have dialogue and invest time to develop a shared understanding of evidence and traits

– Employ observable behavioral checklists rather than traditional Likert scales with inherently wide variability.

17

TRAIT Good Satisfactory Not Acceptable

Recognizes own personal biases that can influence decision making outcomes.

States assumptions and identifies and clarifies personal beliefs that may affect decision outcomes

States assumptions and identifies but does not clarify personal beliefs that may affect decision outcomes

Does not state assumptions or does not identify personal beliefs that may affect decision outcomes

Learns from history by including a discussion from the past where managers faced similar ethical issues.

Demonstrates a good appreciation for prior history where managers faced similar ethical issues

Demonstrates a fair appreciation for prior history where managers faced similar ethical issues

Does not include a discussion of prior history where managers faced similar ethical issues

Limits the expression of self-interest and other outcomes of marketplace logic.

Demonstrates substantive constraint in the expression of self interest and other outcomes of market-place logic

Demonstrates some constraint in the expression of self interest and other outcomes of market-place logic

Demonstrates no constraint in the expression of self interest and other outcomes of market-place logic

Abstains from the tendency to justify the means by virtue of the end.

Clearly abstains from tendency to justify the means by virtue ends. Issues are clearly considered and decision-making is mindful of this tendency.

Shows awareness of the tendency to justify the means by virtue of the end but actions to avoid the tendency are not very deliberate

Does not abstain from the tendency to justify the means by virtue of the end

Adapted from Fogarty 2005

18

A Hypothetical Summary of Evidence Based on a Standardized Rubric

TRAIT N Good Satisfactory Not Acceptable


200 30(15%)

160(80%)

10(5%)


200 22(11%)

88(44%)

90(45%)


200 40(20%)

100(50%)

60(30%)


200 36(18%)

160(80%)

4(2%)

Average 16% 64% 21%

19

The assessor’s dilemma and course-embedded assessment

Observation 3: – Assessment is useful and necessary – Assessment requires resources, commitment, and attention

At many schools, students complete a battery of assessment center activities as part of a required course (Bommer, Rubin & Bartles 2005).

Rockhurst University invested significant sums to create an infrastructure to support course embedded assessment (Bassett, Daley, Haefele 2005).

Many schools administer assessment activities in the capstone course (e.g., Business Strategy & Policy courses at The University of Akron)

Surveys used by many schools are costly and generate reams of data (e.g., EBI)

Assessment evidence gathering, analysis and reporting require significant time commitments

Accounting departments spent less than $5,000 in 2000 and 60% said they had no resources for assessment in 2004 (Hindi & Miller 2000; Calderon et al. 2004)

Assessors (faculty) need time for other valued activities

20

The assessor’s dilemma and course embedded assessment

Specialized assessment activities (e.g., assessment center, assessment day, special tests) …Generally more overhead

UA uses the following in-house tests:• The CCT (core curriculum test)• Major field tests in finance• Major field test in accounting • Major field test in marketing

Attitudes & Opinions - Surveys, interviews, focus groups ………Generally more overhead

UA uses the following:• EBI surveys• Graduating students exit surveys• Employer surveys• Alumni surveys• Advisory board reviews

Course-embedded assessment (done as part of regular, on-going activities in a course) …………. Generally less overhead

UA uses the following:• “Management Projects”• Cases, simulations and projects• Writing samples and writing

portfolios

DIRECT INDIRECT

21

The assessor’s dilemma and course embedded assessment

Lesson 3: Course-embedded assessment (CEA) transforms the evidence collection process from a potentially burdensome activity to one that is part of normal classroom activity

– Create a support infrastructure for CEA Consensus, Classroom Activities, Training, Instruments, & Medium for Reporting and Dialogue (CATIM)

– Focus on program learning goals rather than learning goals for a specific course

– Make achievement of program learning goals public– Use incentives to generate faculty interest in CEA – e.g.,

Rockhurst

22

Data reporting and active engagement vs. data hoarding and passive compliance

Observation 4– Close to 75 percent of business school deans report that

their faculty discuss assessment data regularly at faculty meetings.

– Most do so at least once each academic year.– USAF Academy reports data to faculty regularly and have

dialogue on how data compare with expectations– Seton Hall and Ohio Northern prepare an annual

assessment report for faculty – Kings College sets aside an Assessment Day for faculty to

make presentations on assessment results.

23


How programs report results?Percent (N=105)

Faculty retreat 26

Regular faculty meetings 69

Departmental meetings 44

Committee meetings 44

Oral reports 25

Written reports 50

Special faculty meeting 14

N=160 vs. N=105? Implications

24


Lesson 4: Assessment reports are the basis for dialogue, reflection, and proposals for change– Offer relatively seamless opportunities for viewing

and reviewing reports (e.g., WebCT)– Discuss reports regularly– Identify opportunities for improvement– Create a set of action items after discussions

25

A Hypothetical Summary of Evidence Based on a Standardized Rubric

TRAIT N Good Satisfactory Not Acceptable


200 30(15%)

160(80%)

10(5%)


200 22(11%)

88(44%)

90(45%)


200 40(20%)

100(50%)

60(30%)


200 36(18%)

160(80%)

4(2%)

Average 16% 64% 21%

26

Visualization and improvement opportunities

Radar Chart of Item Analysis

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

80.00%

90.00%

100.00%

12

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

1920

2122

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

3940

Observation 5: Some schools use innovative charts and graphs to focus faculty attention (e.g., Redle & Calderon 2005)

27

Trends in U of Akron’s Major Test in Finance

Trend in Mean and Median Student Scores

40.00%

45.00%

50.00%

55.00%

60.00%

65.00%

70.00%

75.00%

80.00%

Test date

28

Domain Objectives

59.76%

56.19%

74.09%

54.39%

69.86%

61.77%

76.12%

78.70%

51.87%

50%

55%

60%

65%

70%

75%

80%

Through Fall 2001 During 2002 Through Fall 2002 Through Spring 2004 Through Fall 2004 Through Spring 2005

29

Lesson 5: Attention directing graphs and charts lead to more focused assessment dialogue – Oriented toward problem-solving– Focus may include:

Assessment method Faculty development Student Classroom teaching and learning methods Teaching resources and the physical infrastructure for learning Learning goals and curriculum alignment

Visualization and improvement opportunities

30

Closing the Loop

Percent (N=143)

No 42

Yes 58

Total 100

Observation 6:

Has assessment led to significant curriculum changes?

31

Closing the loop

36% of business schools report that they have a formal written assessment plan

Schools with a formal plan are more likely than those without to report success in using assessment to improve the curriculum (Chi-square=4.77, DF=1, p-value=.02)

32

Closing the loop

Lesson 6: Plan before you leap!!!– Powerful tool for effective management and communication of

assessment activities. – Guides schools through each major assessment activity – A basic plan could be designed as a grid that includes columns

for: program learning objectives assessment methods to be used in collecting evidence about student

learning for each objective identification of the measurement metrics and expectations for each

learning goal a procedure for administering the assessment activity a time line for implementation List of persons responsible for coordinating data collection

33

Assessment Plan – A Basic ExampleStudents will: Method used/

EvidenceExpectations Assessment Procedure Primary

Responsibility of

Demonstrate knowledge and understanding of core accounting fundamentals (financial, cost, systems, auditing, taxation)

Method: Major Field Test in Accounting (MFTA) ---(75 MC questions)

Evidence: Raw percentage scores on the MFTA

- Satisfactory value-added (≥45%) - Satisfactory raw scores on the post-test (≥70%)- Satisfactory achievement on each cognitive learning goals

- Computerized Pre/Post Test. - Pre-test administered in first accounting course-Post-test administered in accounting capstone - Consistent incentives and awards (e.g., 5% of grade in capstone)

Department Chair

Demonstrate ability to apply core accounting fundamentals through case analyses and projects.

Method: Course-embedded case/project to demonstrate ability to apply accounting fundamentals

Evidence: Ratings for individual students based on standardized rubric developed for the program by the faculty

- At least satisfactory rating on all traits in the standardized rubric- Overall, 70% perform at least at the satisfactory level

- Relatively short case/project that span multiple areas in accounting- Students analyze cases or complete project as part of requirements capstone.- Case analyzes/project are rated by instructor and a panel of professionals

Faculty who teach the accounting capstone course

34

Conclusion

Assessment is effective if it leads to congruency, alignment and change

Assessment must be planned, supported, and executed in a systematic manner

Assessment is a systematic and deliberate extension of what faculty normally do

Keep what you do simple Avoid fixations!

Documents

1 Assurance of Learning in Business Schools: Observations, Implementation Issues & Guidance Thomas G. Calderon Chair & Professor of Accounting/ Director,