College of Engineering Outcome Assessment: A Seven-Year Study Professor David G. Meyer School of Electrical and Computer Engineering

College of Engineering

Outcome Assessment: A Seven-Year Study

Professor David G. Meyer

School of Electrical and Computer Engineering


Introduction

• A proven method for satisfying the Accreditation Board for Engineering and Technology (ABET) “Criterion 3” requirements is the formulation of outcomes specific to “core” courses in a curriculum, which are tied to the program outcomes.

• This presentation summarizes a 7-year study on formulating and assessing course-specific outcomes


Basic Questions Addressed

• Formulation of content-specific learning outcomes that can be consistently and quantitatively assessed

• Formulation of effective outcome assessment instruments along with mechanisms to determine outcome demonstration thresholds

• Formulation of grading strategies that incorporate outcome demonstration thresholds yet produce results consistent with prior (accepted) grading practices (i.e., course GPA in range of approximately 2.5 – 2.8 on a 4.0 scale)

• Formulation of outcome remediation strategies that are both fair and efficient


Edict from the ABET Visitors

“We want to see evidence of failing grades assigned for cases in which students [who would otherwise be passing but] failed to successfully demonstrate one or more course outcomes.”

Interpretation: We can’t just say that, because a student earned a total score above the “course passing threshold”, that he/she has successfully demonstrated all the course outcomes. Also, application of outcome assessment should produce some form of meaningful and measurable difference.


Major Issues

• Balance– fairness to students

• providing students with a sufficient number of opportunities to demonstrate course outcomes

• ensuring that students’ own work is used as the basis for outcome demonstration success

– overhead for instructor• keeping the incremental workload associated with

outcome assessment and tracking to a minimum• keeping the outcome assessment process

“contained” within a given term/semester


Major Issues

• Evaluation Instruments– exams (whole or question subsets)– quizzes (written/oral)– homework assignments– labs– papers/projects/presentations

• Passing Thresholds– static (what absolute value?)– dynamic (what algorithm?)– consistent with grading cutoff (A-B-C-D) thresholds


Major Issues

• Assigning course grades consistent with proven prior practice, yet reflecting meaningful application of outcome assessment thresholds

• Achieving consistency– semester-to-semester– professor-to-professor– course-to-course– school-to-school– institution-to-institution


Sample Course-Specific Outcomes

ECE 270 – Introduction to Digital System Design1. an ability to analyze static and dynamic behavior of digital circuits2. an ability to represent Boolean functions in standard forms, to map and

minimize them, and to implement them as combinational logic circuits3. an ability to use a hardware description language to specify

combinational logic circuits, including various “building blocks” such as decoders, multiplexers, encoders, and tri-state buffers

4. an ability to design and implement arithmetic logic circuits5. an ability to analyze, design, and implement sequential circuits and use

a hardware description language to specify them 6. an ability to design and implement a simple computer based on

combinational and sequential building blocks

KEY: Modularization of course material


Sample Outcome Assessment ExamPrimary Assessment of Outcome 2 for CN: 0000-X

X 1. Most minimization methods are based on a generalization of…

2. The fact that any combinational logic circuit can be realized using only...

3. The complement of this function is represented by the ON SET...

4. The dual of this function is represented by the ON SET...

5. This expression is an example of...

6. The XOR property listed below that is NOT true is...

7. A NOR gate is logically equivalent to...

8. A circuit consisting of _ is logically equivalent to...

9. This circuit exhibits the following type of hazard...

10. The number of prime implicants is...

11. The number of essential prime implicants is...

12. The number of non-essential prime implicants is...

X 13. The cost of a minimal sum of products realization of this function is...

X 14. The cost of a minimal product of sums realization of this function is...

X 15. The fewest number of 2-input NAND gates needed to realize this function is...

X 16. The fewest number of 2-input NOR gates needed to realize this function is...

17. The fewest number of 2-input OD NAND gates needed to realize this function is...

18. The number of pull-up resistors required is...

19. The simplest realization of the function depicted by this circuit is...

X 20. The minimum SoP function realized by this circuit is...

X 21. The ON-SET of the function realized by this circuit is...

X 22. The minimum PoS function realized by this circuit is...

X 23. Realization of this same function using only 2-input NOR gates would require...

X 24. Steady-state analysis of the function realized by this circuit predicts...

25. The dynamic behavior of the circuit does NOT match the steady-state analysis...

Number of correct answers: 15 (out of 25 questions)


Sample Outcome Demonstration ReportECE 270 Course Outcome Report for: 0000-X, enrolled in Lab Division 5

Outcome 1: An ability to analyze static and dynamic behavior of digital circuits

Outcome 1: Primary Assessment Score = 9/ 25 = 36.0% Passing Threshold = 52.0%

Outcome 1: Sorry - this outcome was not demonstrated

Outcome 2: An ability to represent Boolean functions in standard forms...


Outcome 2: Congratulations - this outcome was successfully demonstrated

Outcome 3: An ability to use a hardware description language to specify combinational...



Outcome 4: An ability to design and implement arithmetic logic circuits



Outcome 5: An ability to analyze, design, and implement sequential circuits...



Outcome 6: An ability to design and implement a simple computer



You will need to retake the outcome(s) you have missed on the Final


Outcome Demonstration Summary ReportECE 270 PRIMARY OUTCOME ASSESSMENT REPORT

Outcome 1 -- Avg Score: 67.4% Threshold: 52.0% Took: 205/206 = 99.5%

Passed: 179/205 = 87.3% Failed: 26/205 = 12.7%


Passed: 167/203 = 82.3% Failed: 36/203 = 17.7%


Passed: 175/203 = 86.2% Failed: 28/203 = 13.8%


Passed: 171/203 = 84.2% Failed: 32/203 = 15.8%


Passed: 171/200 = 85.5% Failed: 29/200 = 14.5%


Passed: 169/200 = 84.5% Failed: 31/200 = 15.5%

Summary of Outcome Demonstration:

Number of students who demonstrated 0 outcome(s) on the primary assessments: 2








Refinements

• Early (discarded) attempts– fixed passing threshold (60%) on weighted sum of

(selected) lab, homework, and exam scores– fixed passing threshold (60%) on primary assessment,

remediation homework, and final assessment

• Main problems– basically everyone “passed” – strategy did not prove to be

effective– impossible to ensure that students’ own work was being

evaluated on labs and (remediation) homework


• Restricting outcome assessment to exams only– fixed passing threshold (60%) on primary and final

assessments– use of “conditional failure” grade for those who would

otherwise be passing, with 3rd attempt for remediation of failed outcome(s) the following semester

• Main problems– fixed passing threshold became a significant factor:

hard (“impossible”) to write exams that produce a predictable mean/distribution

– lack of containment within term (“overhead leakage”)– grade inflation due to allowing final assessment to be

used as replacement score for all N outcomes

Refinements


• Incorporation of dynamic thresholds– used on each primary and final assessment, calculated

as mean – standard deviation, with bounded range of 40% to 60% (choice based on empirical study of data sets in multiple courses over multiple terms)

– must pass at least 50% of N (total) outcomes on the primary assessments to qualify for a passing grade

– final assessment can be used for remediation and/or improvement (replacement) of score on up to N/2 outcomes

– self-contained (no “overhead leakage” to following semester)

Refinements


• Controlling grade inflation– added limited opportunity for grade improvement on

final assessment• effective outcome score = weighted average of

primary and final assessments (60-40)• number of outcomes re-taken limited to half• must have passing grade to qualify for final

– grade inflation associated with grade improvement on final reduced by about an order of magnitude

Refinements


• Reducing outcome-induced failures– students with grades as high as “B” were still failing

course due to failure to demonstrate all course outcomes on two attempts (primary + final)

– award grade of “I” or “E” if otherwise have passing grade after final but still need to satisfy one or more outcomes

– remediation exam(s) taken subsequent semester of enrollment at scheduled exam times with rest of class

– if fail this third remediation attempt on any given outcome, receive failing grade and re-take course

– some “overhead leakage” reintroduced, but minimal

Refinements


ResultsECE270 Outcome Demonstration Success Rate

0

10

20

30

40

50

60

70

80

90

100

4 5 6 7

Trial Number

Per

cen

t S

ucc

essf

ul

PrimaryFinal

Trial 4 – Static thresholds, N replacement, 3 tries

Trial 5 – Dynamic thresholds, N/2 replacement

Trial 6 – Dynamic thresholds, no replacement

Trial 7 – Dynamic thresholds, weighted average


Results





ECE362 Outcome Demonstration Success Rate

0

10

20

30

40

50

60

70

80

90

100

4 5 6 7

Trial Number

Per

cen

t S

ucc

essf

ul

PrimaryFinal


Results





Course Grade Point Average

0

0.5

1

1.5

2

2.5

3

3.5

4

4 5 6 7

Trial Number

GP

A (

4.0

Sca

le)

ECE 270ECE 362

Desired range, based on proven prior practice


Lessons Learned – A “Delicate Balance”

• Assigning course grades consistent with proven prior practice

• Providing incentives for students to successfully demonstrate outcomes, yet not causing grade inflation and/or excessive instructor workload

• Establishing reasonable, meaningful thresholds for outcome demonstration success that are decoupled from “exam difficulty”

• Determining a fair level of pass/fail “filtering” based on outcome demonstration success


Some Remaining Questions

• What is an appropriate amount of outcome-based “filtering” for various kinds of courses? – currently averaging about 10-15% in lower

division courses– is this too much, too little, or about right?

• What is an appropriate balance between number of attempts allowed to demonstrate an outcome, and the incremental workload associated with managing this effort?– currently offer three attempts– is this too much, too little, or about right?


Summary and Conclusions

• What may appear to make the most sense “on paper” may not be the “best” strategy when all factors are considered

• Finding an “optimal” strategy is non-trivial• Different kinds of courses (e.g., “content-

intensive” vs. “design-intensive”) require different outcome assessment instruments, thresholds, and remediation strategies


Senior Design Outcomes and Assessment

1. An ability to apply knowledge obtained in earlier coursework and to obtain new knowledge necessary to design and test a system, component, or process to meet desired needs – design component report

2. An understanding of the engineering design process – individual lab notebook

3. An ability to function on a multidisciplinary team – project specific success criteria (functionality)

4. An awareness of professional and ethical responsibility – professional component report

5. An ability to communicate effectively, in both oral and written form – formal design review, final written report, and final presentation


Outcome Tracking Results forDigital Systems Senior Design

Outcome 1 – design component report

Outcome 2 – individual lab notebook

Outcome 3 – project specific success criteria

Outcome 4 – professional component report

Outcome 5 – technical communication skills

2 Tablet PCs per team

1 Tablet PC per team

TCSP sessions initiated

HTML notebooks

initiated


Summary and Conclusions

• Have the “basic questions” been answered?

– Formulation of content-specific learning outcomes that can be consistently and quantitatively assessed YES

– Formulation of effective outcome assessment instruments along with mechanisms to determine outcome demonstration thresholds YES

– Formulation of grading strategies that incorporate outcome demonstration thresholds yet produce results consistent with prior (accepted) grading practices (i.e., course GPA in range of approximately 2.5 – 2.8 on a 4.0 scale) YES

– Formulation of outcome remediation strategies that are both fair and reasonably efficient YES


Discussion Questions

Should outcome assessment strategies be more widely deployed across the ECE curriculum?

What are some successful assessment methods used in industry that could translate to the academic environment?

Documents

College of Engineering Outcome Assessment: A Seven-Year Study Professor David G. Meyer School of Electrical and Computer Engineering