Outcomes Assessment Using Capstone Experiences

Outcomes Assessment Using Capstone Experiences

Joseph A. ShaeiwitzChemical Engineering Department

West Virginia UniversityP.O. Box 6102

Morgantown, WV 26506-6102304-293-2111 ext. 2410

[email protected]

http://www.che.cemr.wvu.edu/ugrad/outcomes/

Assessment Survey

Please complete the pre-workshop assessment survey

Outline

Introduction/MotivationHow to include as many

outcomes as possible in a capstone experience

Desired attributes of capstone experiences

Outline (cont’d) Development of rubrics to assess

capstone experiences Simultaneous assessment and

grading Using oral presentations for

assessment of capstone experiences

Summary

Outline




Motivation Self-assessment measures such as

surveys are insufficient, assessment by the faculty is needed

All engineering programs already have capstone experiences

Students are expected to apply previously learned knowledge in capstone experiences

oneclass

oneclass

education process

one course

one course

enteringcollege

graduate alumnus

Feedback Model

Outline




Required Outcomes Apply math, sci,

engr Experiments Design Teams Solve engr problems Ethics

Communication Broad education Life-long learning Contemp. Issues Modern tools

Exercise 1 Suggest a capstone experience

that includes as many required outcomes as is possible

Focus on what needs to be added to commonly used experiences to include additional outcomes

Some Ideas Comprehensive individual/group design

project Includes all technical aspects Include material not taught in class Can include safety, life-cycle analysis,

pollution prevention, environmental impact statement

Requires use of specialized, sophisticated software

Outline




Attributes of Capstone Experiences

What attributes do you expect to see in your capstone experiences?

What skills and/or knowledge do you expect to be demonstrated in capstone experiences?

Exercise 2 List as many attributes as possible

for the capstone experience described in Exercise 1

Focus on what you would be evaluating when grading the final project report

Some Ideas Writing – grammar, punctuation,

format, good technical explanations, logical development

Oral – use of visual aids, organization, delivery mechanics

Technical – apply math, science, engineering; design of equipment; optimization based on economics

Outline Development of rubrics to assess




Summary

Development of RubricsWhat attributes do you expect

to see in your capstone experiences?

What skills and/or knowledge do you expect to be demonstrated in capstone experiences?

Development of Rubrics (cont’d)

How would you describe excellence for each attribute?

What is not acceptable for each attribute?

Fill in intermediate descriptions

Four-Point Scale for Rubrics

4 – exceeds expectations

3 – meets expectations

2 – below expectations

1 – not acceptable

Use of Rubrics

Oral Reports

Written Reports

Technical Content

Advantages of Method

Can involve many faculty members

Standardization of evaluation Is done while gradingCan be adapted to guideline

for grading

Exercise 3

For the attributes listed in Exercise 2, develop a rubric for evaluation of these attributes

Attribute 1-Not acceptable 2-Below

expectations 3-Meets expectations

4-Exceeds expectations

Score

Report Mechanics Organization inappropriate content of

most sections of report some content in inappropriate section of report

content appropriate to all section of report

unique organization enhances readability and/or understandability of report

Complete Story Told no story told, very incomplete

aspects of complete story missing

complete story told additional material enhances quality of report

Aesthetics unacceptable – e.g., tables and figures cannot be read/understood, fonts difficult to read

some portions are sloppy and difficulty to read

text, tables, figures readable and understandable

text, tables, figures so clear and understandable as to enhance report impact

Format so many format errors as to make report useless

some format errors format followed unique format aspects that enhance report impact

Spelling any spelling errors only spelling errors are different spellings for same pronunciation

no spelling errors

Grammar and Punctuation too many grammar and punctuation errors

grammar and punctuation errors

only a very few grammar or punctuation errors

no grammar or punctuation errors

Length far too long or too short

too long or too short

appropriate length

Attribute 1-Not acceptable 2-Below

expectations 3-Meets expectations


Score

Effective use of Visual Aids (VA)

Clarity and readability not clear or readable difficulty reading clear and readable superior clarity and readability

Use of space on VA VA unreadable because too crowded

too little or too much information of VA

appropriate amount of information on VA

VAs very well laid out

Lettering readable font unreadable font too small font readable Color, over- or under-use (if used)

colors too hard to distinguish, colors do not project well

poor choice and use of colors

primary/easily distinguishable colors

use of color enhances clarity of presentation

Wording concise slides full of text slides too wordy slides appropriate Appropriate amount of information per VA

so much information per VA or so much missing information to make VA useless

too much information per VA, missing information such as size of total pie

appropriate level of information per slide

Presentation Organization Logical order of topics totally disjointed, no

organization some items presented out of order

organization as per guidelines

superior organization enhances communication

Appropriate use of time: Not too long /short

far too long or far too short

somewhat too long or too short

appropriate length

Complete "story" told story missing, no story told

story incomplete complete story told

Introduction: Problem stated problem not stated problem poorly stated

problem clearly stated problem clearly stated, good perspective on problem shown

Attribute 1-Not acceptable 2-Below expectations

3-Meets expectations


Score

Design of equipment, Analysis of performance of existing equipment, Understand interrelationship between equipment in process

Design of individual equipment

major errors in individual equipment design

some errors in equipment design

equipment designed correctly

unique aspects of equipment design enhance result

Understand interrelationship between equipment on flowsheet

no understanding of equipment interrelationship

minimum understanding of equipment interrelationship

clear understanding of equipment interrelationship

exploitation of equipment interrelationship to enhance result

Constraints/limitations of individual equipment and flowsheet understood

constraints/ limitations not understood

not all constraints/ limitations understood

constraints/ limitations clearly understood

exploitation of constraints/ limitations to enhance result

Response to questions indicates understanding of ChE principles

response to questions demonstrates lack of understanding

response to questions shows gaps in understanding

response to questions shows clear understanding

response to questions shows superior understanding

Significance of conclusions understood

lack of understanding gaps in understanding clear understanding superior understanding

Apply chemistry, math, physics, engineering science

Apply engineering science inability to apply principles weak application of principles

good application of principles

superior application of principles

Response to questions indicates ability to apply these principles

response to questions demonstrates lack of ability to apply these principles

response to questions shows gaps in ability to apply these principles

response to questions shows clear ability to apply these principles

response to questions shows superior ability to apply these principles

Scoring Sheet for all Design Projects

Technical Content Design of equipment, analysis of performance of existing equipment, understand interrelationship between equipment in process

Apply chemistry, math, physics, engineering science Resolve complex problem into components Apply economic, physical constraints and optimization methods to obtain solution

Computer usage Application of safety principles Other economic, global, societal, and legal considerations

Oral Presentation

Effective use of visual aids Presentation organization Effective use of software to prepare visual aids Group synergism/dynamics Presentation mechanics Response to questions

Written Report

Report mechanics Content Effective use of software to prepare written report





Summary

Primary Trait Analysis Rubric-like grading guide Converts “unstated criteria” (e.g., “it

feels like a B,” to “highly explicit criteria.”

Converts from norm-referenced to criterion-referenced grading

B. E. Walvoord and V. J. Anderson, Effective Grading. A Tool for Learning and Assessment, Jossey-Bass, 1998.

Our Experiences Five-point scale is difficult to develop We use four-point scale shown earlier Approximate grades

• A = several 4, all others 3• B = all 3• C = average about 2.5 or so• D = average around 2• F = average well below 2

Large-Group Design Project Students work under direction of

student chief engineer Faculty members play roles – one is

client, other is student company vice president

Evaluations like in company Rubric developed for grading

Participation (35%)

F D C B A

No attendance Did not

complete assignment and displayed poor attitude

No participation in group meetings

Missed 5 or more group meetings

Turned in assignments sporadically

Displayed a poor attitude during group activities

Missed 3 or more group meetings

Turned in late assignments

Was not task oriented during group meetings

Attended all group meetings

Completed all assignments

Worked well with the group during meetings

Attended all group meetings

Completes all assignments with an excellent attitude

Performed extra tasks that produced new insight within the group

Technical Quality (40%)

F D C B A

No understanding & synthesis of research

The ideas were not expressed clearly

Major errors throughout design calculations

Poor understanding & synthesis of research

The ideas were not expressed clearly

Many minor errors in calculations

Average understanding & synthesis of research

The ideas were expressed clearly

Few errors in design calculations

Good understanding & synthesis of research

The ideas were expressed clearly

No errors in design calculations

Excellent understanding & synthesis of research

Ideas exceeded expectations and were expressed clearly

No errors in design calculations

Communication (25%)

F D C B A





Summary

Feedback from Q & A Session

Have presentation followed by question and answer session

Students get immediate feedback Ask questions and follow-up

questions Look for patterns – common errors

or common good ideas

Types of Questions Asked Can they explain what was done

and why? Quiz students on related or

peripheral concepts Error is observed – convert

session into tutorial How has student applied what

was learned?

Information Seeking

Level of understanding and ability to communicate it

Was “right” answer obtained for wrong reason or accidentally?

Is “big picture” understood? How was solution obtained?

Example – weak student Student chooses reactor

temperature of 325°C Question: Why was this reactor

temperature chosen? Good: Because it was economic

optimum Bad: I just chose it

Example – better student Student chooses reactor temperature

of 325°C and shows graph illustrating that it is economic optimum

Question: Why is this optimum? Why are economics less favorable above and below this temperature?

Good: Provides explanation Bad: I just tried a range of

temperatures and this is what I found.

Example – very good student Student chooses reactor temperature

of 325°C, shows graph illustrating that it is economic optimum, and explains why

Question: Did you investigate other variables such as pressure, conversion?

Good: Yes, and explains result Bad: No, I just investigated

temperature.

Example – excellent student

Student chooses reactor temperature of 325°C, shows graph illustrating that it is economic optimum, explains why, and does same for other decision variables

Question: It becomes more difficult to find good questions

Exercise 4

Ask me questions about this presentation aimed at revealing my understanding of assessment, ABET EC 2000, rubric development, etc.





Summary

Summary Assessment using capstone

experiences can complement surveys – assessment by the faculty

It is possible to include many outcomes in capstone experiences

Identify desired attributes of capstone experiences

Summary (cont’d) Can develop rubrics to assess

capstone experiences Can use primary trait analysis for

simultaneous assessment and grading

Can use oral presentations as part of assessment of capstone experiences

Web Site

http://www.che.cemr.wvu.edu/ugrad/outcomes/

Assessment Survey

Please complete the post-workshop assessment survey

Documents

Outcomes Assessment Using Capstone Experiences