1

In Search of Closing the Continuous In Search of Closing the Continuous Improvement Loop: Outcomes Improvement Loop: Outcomes

Assessment of Student KnowledgeAssessment of Student Knowledge

M. Suzanne ClintonAssociate Dean, School of Business

Lisa K. WolfeMBA Graduate Coordinator

2

Assessment• Identify what we should do• Measure the quality and effectiveness of

our work• Document what we are doing• Are we doing what we should be doing?

(Advisory Board)• Plan how to IMPROVE what we do• A five-step continuous process

IESIESIESIES

3

Step One

• Develop common syllabi for courses taught by various faculty

• Identify approximately ten core competencies for each course

• A core competency is the knowledge that a student must acquire to be able to complete the course successfully

4

Step Two

• Match core competencies to specific exam questions, writing assignments, projects, etc.

• Set up spreadsheets to simplify data entry

• Establish appropriate benchmarks

5

Step Three

• Collect Data– Perceived knowledge– Actual knowledge

• Compare Perceived and Actual measures to benchmark

• Document, document, document…

6

Step Four

• Evaluate comparisons to determine – If changes are needed– What specific changes are needed

• Document planned changes• Implement changes• Document actual changes• Document results (this takes us back to

Step Three)

7

Step Three REVISITED

• Collect Data AGAIN– Perceived knowledge– Actual knowledge

• Compare Perceived and Actual measures to benchmark AGAIN

• Document, document, document…

8

Step Four REVISITED

• AGAIN, evaluate comparisons to determine – If changes are needed– What specific changes are needed

• Document planned changes AGAIN• Implement changes AGAIN• Document actual changes AGAIN

9

Step Five• Repeat Steps Three and Four each time a

course is offered• Develop trend lines• Periodically review syllabi and core

competencies• Review syllabi and core competencies with

Advisory Board• As exams, written assignments and projects

are revised, so too will be the core competencies and their measures

• The entire process is continuous

10

Examples of Core Competencies for Statistics

Core Competency #1

The student will be able to calculate the mean, median, mode, standard deviation and variance

Core Competency #2

The student will be able to calculate relative frequency, conditional probability and joint probability

Core Competency #3

The student will be able to calculate permutations and combinations and be able to apply Bayes’ Theorem

Core Competency #4

The student will be able to calculate probabilities associated with binomial distributions, Poisson distributions and the hypergeometric distribution

11

Examples of Perceived Survey1. Your ability to calculate mean, median, mode, standard deviation

and variance.No Understand

Understanding /____/____/____/____/____/____/____/ Very well

1 2 3 4 5 6 7

2. The student will be able to calculate relative frequency,

conditional probability and joint probability No Understand

Understanding /____/____/____/____/____/____/____/ Very well

1 2 3 4 5 6 7

3. The student will be able to calculate permutations and combinations and be able to apply Bayes’ Theorem.

No Understand

Understanding /____/____/____/____/____/____/____/ Very well

1 2 3 4 5 6 7

4. The student will be able to calculate probabilities associated with binomial distributions, Poisson distributions and the hypergeometric distribution

No Understand

Understanding /____/____/____/____/____/____/____/ Very well

1 2 3 4 5 6 7

12

Sample Test Questions forCore Competency #1

X P(X)

0 .1

1 .3

2 .4

3 .2

1. Calculate the mean of this distribution.

2. Calculate the standard deviation of this distribution.

13

Sample Test Questions for Core Competency #2

1. Find the entire area under the standard normal curve (the Z curve) which lies to the left of

Z = -0.21

2. Find the probability that a potential investor will purchase shares in at least one of the two funds.

3. What is the probability that a company employee believes that he/she would be financially secure if he/she lost his/her job?

14

Sample Test Questions for Core Competency #3

1. How many combinations of 3 can be chosen from 75?

2. A busy executive has to meet with five production managers during the day. The executive needs to decide in which order to see the managers. How many different orderings can the executive choose?

15

Sample Test Questions for Core Competency #4

1. In a shipment of 15 room air conditioners, there are 4 with defective thermostats. You are going to randomly inspect 5 of the air conditioners in this shipment. What is the probability that of the 5 air conditioners you choose to inspect 3 are good and 2 are defective?

2. Make a continuity correction and use the normal approximation to the binomial to determine P(X = 50). Suppose that X has a binomial distribution with n = 100 and p = .5.

16

Sample of Core Competency Calculations

Core Competency #1

Core Competency #2

Exam # & Question # E1 Q1 E1 Q2 E1 Q3 E1 Q4

Max. Points 5 5 10 10

Student #1 5 5 10 10

Student #2 5 5 10 0

Student #3 5 5 10 10

Student #4 0 5 10 10

Total points 15 20 40 30

Total points/# of students 3.7 5 10 7.5

Average Score by Question 75% 100% 100% 75%

Average Score by CC 88% 88%

17

Program Objectives

• Program Objectives – educational purposes developed from an analysis of consumer needs– Based on Cameron’s mission– Clearly stated– Relatively stable over time (2-3 years)– Operationalized (put in measurable terms)– Basis for program planning and strategy development– Faculty expectations about what is to be learned– Criteria for program evaluation– Basis for curriculum development and faculty selection and

development

18

Program Strategies

• Describe how program requirements ensure student learning experiences

• Clearly state which program components serve which program objectives

• Ensures appropriate attention is given to each objective

• Assessment results are used annually to evaluate and revise program strategies

19

Assessment Strategies

• Measurement options to be used for each objective (exam, group assignment, etc.)

• Students who are targeted• Time lines for data collection and analysis• Methods of analysis to be used on the assessment

data• Decision criteria to be used to formulate

conclusions from the results of the analyses

20

Assessment Model

• Essential part is the evaluative conclusions regarding student strengths and weaknesses related to program strengths and weaknesses

• Assessment results are used annually to evaluate each data collection and analysis method

• Appropriate modifications are included in the assessment plan for the next cycle

21

Display of Assessment Data

• Display the tables and narrative to describe assessment measures taken and number of students participating

• Measures must be reliable and valid• Measures should tie directly to program

objectives

22

Analysis of Achievement of Objectives Assessed

• Describe how the assessment data indicated program quality

• Use multiple measures of program outcomes

• Discussion of instructional changes that occurred and their impact on the quality of the program should be included

23

Action Plan

• List specific actions to be taken as a result of the assessment process

• Indicate – Modifications proposed– Time line for implementing changes– Resource implications of the changes

• Changes submitted to curriculum committee must be supported with assessment data

• Identify instructional changes that will improve program outcomes

24

Conclusion

• Discussion• Questions

Thank you for coming!