Direct vs Indirect Assessment of Student Learning: An Introduction

Dr. Sheila Handy, Chair Business Management and Co-Chair University Assessment Committee

Sponsored by University Assessment Committee

February 24, 2014

Assessment• “The most important purpose of assessment should be not

improvement or accountability but their common aim: Everyone wants students to get the best possible education. Everyone wants them to learn what’s most important. A college’s mission statement and goals are essentially promises that the college is making to its students, their families, employers, and society. Today’s world needs people with the attributes we promise. We need skilled writers, thinkers, problem-solvers and leaders. We need people who are prepared to act ethically, to help those in need, and to participate meaningfully in an increasingly diverse and global society. Imagine what the world would be like if every one of our graduates achieved the goals we promise them! We need people with those traits, and we need them now. Assessment is simply a vital tool to help us make sure we fulfill the crucial promises we make to our students and society. “

Linda Suskie, 2010

Continuous Improvement of Student Learning

Setting the Stage

•Middle States Commission on Higher Education

• “…each program should employ direct measures of student learning…”

• “…each stated learning outcome should be aligned to a measure, a performance goal, and a finding. Measures can be used to evaluate more than a single learning outcome.”

Assessment Measures - Definitions

Direct Indirect

•Direct examination or observation of student knowledge, skills, attitudes or behaviors to provide evidence of learning outcomes.

•Perceived extent or value of learning experiences

Assessment Measures - Examples

Direct Indirect

•Exam▫Standardized▫Locally developed▫Embedded questions

•Juried review•Portfolios with rubrics

•Surveys▫Students▫Alumni

•Focus Groups•Student Records▫Class Attendance

Short Table Activity

• Introductions

•Envelopes▫Match Assessment Measure and Description▫Sort into Direct and Indirect Measures

Standardized Tests

•Licensing Exams▫Education - Praxis, PAPA▫Nursing - NCLEX▫Accounting - CPA Exam

•Major Field Test▫Developed by ETS

▫Administered in a proctored environment as paper and pencil or online

MFT subjects▫Biology▫Business▫Chemistry▫Computer Science▫Criminal Justice▫Literature in English

▫Math▫Physics▫Political Science▫Psychology▫Sociology

Standardized TestsAdvantages Disadvantages

Carefully developed Highly reliable Professionally scored Nationally normed

▫ If tests don’t match LOs of program, scores will be low

▫ MC questions that test facts – faculty may focus on higher order skills

▫ May be expensive▫ Students may not take

seriously

Locally Developed ExamAdvantages Disadvantages

• Can be tied-in to specific LOs• Can be administered as part of

regular semester testing• Can use publisher developed

questions

• Reliability and generalizability not known

• Time required to prepare questions

• Scoring may take a long time• Norms not available

Embedded QuestionsAdvantages Disadvantages

• Can be conducted as part of regular student testing

• Does not require collection of additional data

• Can be used for both grading and assessment

• Faculty may think they are being assessed

• Takes time to develop and grade

Embedded Questions

•One way to assess learning without faculty “angst”▫Add assessment questions as a quiz administered

at the beginning of the next course in the sequence

▫Effective if different faculty teach multiple sections of an introductory course

Juried Review

•Suitable for assessing ▫Performing arts projects▫Written assignments▫Oral presentations

Juried ReviewAdvantages Disadvantages

• Provides experience and practice for student performers

• Can be formative and summative – indicating “value added”

• Can also introduce students to the standards they will be expected to achieve

• Achieving consensus with regard to the evaluation rubric

• May not be sufficient to determine if students have achieved the LOs set by the faculty

Portfolios With Rubrics

•Suitable for many majors▫Art▫English▫Computer Science▫Business▫Education

Portfolios With RubricsAdvantages Disadvantages

• Provides students with evidence to submit to potential employers

• Tracks student work over time• Can assess higher levels of

learning• Allows students to reflect on

their learning

• Faculty must agree on assessment rubric

• Time consuming• Storage of evidence

Case Study

Your goal is to develop a plan to assess this student learning outcome for the Engineering Program for the upcoming academic year. This plan should include two direct measures.

Wrap Up

•Resources

▫Materials in Folders

▫University Assessment Committee

▫Assessment Consulting Team

▫Website