Assessment

Is like a dancer’s mirror.

It improves one’s ability to see and improve one’s performance.

Alexander Astin

1993

ASSESSMENT . . .

“a rich conversation

about student learning

informed by data.”

-- Ted Marchese --

AAHE

Assessment of Individual Student Development

Assessment of basic skills for use in advising• Placement• Counseling

Periodic review of performance with detailed feedback

End-of-program certification of competence• Licensing exams• External examiners• CLAST

Key Results of Individual Assessment

Faculty can assign grades

Students learn their own strengths and weaknesses

Students become self-assessors

A Second Look

Across students

Across sections

Across courses

Where is learning satisfactory?

What needs to be retaught?

Which approaches produce the most learning for which students?

Group Assessment Activities

• Classroom assignments, tests, projects

• Questionnaires for students, graduates, employers

• Interviews, focus groups• Program completion and placement• Awards/recognition for graduates• Monitoring of success in graduate

school• Monitoring of success on the job

Use of Results of Group Assessment

• Program improvement

• Institutional and / or state peer review

• Regional and / or national accreditation

Outcomes Assessment

The process of providing credible

evidence of the processes

and outcomes of higher education

undertaken for the purpose of

improving programs and services

within the institution.

Banta, T. W.

Some Purposes of Assessment

1. Students learn content

2. Students assess own strengths

3. Faculty improve instruction

4. Institutions improve programs/services

5. Institutions demonstrate accountability

Assessment of Learning

1. Faculty agreement on outcomes and performance standards

2. Evidence in syllabi and assignments that outcomes are taught

3. Collective faculty review of group performance

4. Use of findings to improve instruction and curriculum

Most Faculty Are Not Trained as Teachers

Faculty Development

Can Help Instructors: Write clear objectives for student learning in

courses and curricula Individualize instruction using a variety of

methods and materials Ask questions that make students active

learners Develop assessment tools that test higher

order intellectual skills

Taxonomy of Educational Objectives

(Bloom and Others, 1956)Cognitive domain

categories

Knowledge

Comprehension

Application

Analysis

Synthesis

Evaluation

Sample verbs for outcomes

Identifies, defines, describes

Explains, summarizes, classifies

Demonstrates, computes, solves

Differentiates, diagrams, estimates

Creates, formulates, revises

Criticizes, compares, concludes

Organizing for Assessment

Goal Course Measure Findings Uses

Write Portfolio

Speak Speech

Think Test

Find Information

Project

Planning for Learning and Assessment

1. Whatgeneraloutcomeare youseeking?

2. Howwould youknow it(theoutcome)if you sawit? (Whatwill thestudentknow orbe able todo?)

3. How willyou helpstudentslearn it?(in classor out ofclass)

4. How couldyoumeasureeach of thedesiredbehaviorslisted in #2?

5. What aretheassess-mentfindings?

6. Whatimprove-mentsmight bebased onassess-mentfindings?

Faculty and Staff Development

Focus faculty and student affairs professionals on improving learning in and outside class

Attend conferences together Study literature on student learning Provide workshops on teaching and learning Provide resources (e.g., grants, summer

salary, release time)

Direct Measures of LearningAssignments, exams, projects, papers

Indirect MeasuresQuestionnaires, inventories, interviews

- Did the course cover these objectives?- How much did your knowledge increase?- Did the teaching method(s) help you

learn?- Did the assignments help you learn?

Select or DesignAssessment Methods

1. Match with goals2. Use multiple methods3. Combine direct and indirect measures4. Combine qualitative and quantitative

measures5. Consider pre - post design to assess

gains6. Use built-in points of contact with

students

Outcomes Assessment Requires Collaboration

In setting expected program outcomes

In developing sequence of learning experiences (curriculum)

In choosing measures

In interpreting assessment findings

In making responsive improvements

Barriers to Collaboration in the Academy

1 Graduate schools prepare specialists2 Departments hire specialists3 Much of our scholarship is conducted

alone4 Promotion and tenure favor individual

achievements -- interdisciplinary work is harder to evaluate

Change the Focusfrom

TEACHING

to

LEARNING

Barr & Tagg 1995

Functions of the Work of the Professoriate

Scholarship of Discovery

Scholarship of Integration

Scholarship of Application

Scholarship of Teaching

Ernest Boyer - 1990

Good assessment is good research . . .

An important question An approach to answer the

question Data collection Analysis Report

-Gary R. Pike (2000)

Campus Interest in Assessment

WHAT WORKS in….

increasing student retention? general education? use of technology in instruction? curriculum in the major?

Measures of Critical Thinking

1. Academic Profile (ETS)

2. Collegiate Assessment of Academic Proficiency (ACT)

3. California CT Dispositions Inventory (Faciones)

4. California CT Skills Test (P. Facione)

5. Cornell CT Test (Ennis & Millman)

6. Tasks in CT (ETS)

7. Reflective Judgment Inventory (King & Kitchener)

8. Watson Glaser CT Appraisal (Psych Corp)

Collegiate Assessment of Academic Proficiency

(CAAP from ACT)

Individual ModulesReadingWriting (MC and Essay)MathematicsScience ReasoningCritical Thinking

COLLEGE BASE

InterpretiveReasoning

StrategicReasoning

AdaptiveReasoning

Competences

SUBJECTS

English

Math

Science

SocialStudies

EnglishSubject Clusters

InterpretiveReasoning

Writing Skills

Reading/Literature SkillsReading Critically

Reading AnalyticallyUnderstanding Literature

Are Standardized Tests the Answer?

Not available in many fieldsDo not measure all that is taughtUsually assess knowledge, not

performanceMay be standardized on

unrepresentative norm groupProvide few, if any, subscoresDo not indicate why scores are low

Start with Measures You Have

Assignments in coursesCourse examsWork performanceRecords of progress through

the curriculum

Primary Trait Scoring

Assigns scores to attributes (traits) of a task

STEPS Identify traits necessary for success in

assignment Compose scale or rubric giving clear

definition to each point Grade using the rubric

Can Develop a Research Paper

1. Narrows and defines topic

2. Produces bibliography

3. Develops outline

4. Produces first draft

5. Produces final draft

6. Presents oral defense

  

 

Out-standing

Accept-able

Unaccept-able

BibliographyOutstanding – References current,

appropriately cited, representative, relevant

Acceptable – References mostly current, few citation errors, coverage adequate, mostly relevant

Unacceptable – No references or containing many errors in citation format, inadequate coverage or irrelevant

Mapping Course Outcomes to Program Outcomes

Outcomes Course 1 Course 2 Course 3

1 2

3

4

5

6

7

Sophomore Competence in Mathematics(Multiple choice responses & supporting work)

Score3

2

1

0

CriterionClear conceptual understanding, consistent

notation, logical formulation, complete solution

Adequate understanding, careless errors, some logic missing, incomplete solution

Inadequate understanding, procedural errors, logical steps missing, poor or no response

Problem not attempted or conceptual understanding totally lacking

Ball State University

Journal Evaluation

1. Entries accurately and vividly record objective observations of site experiences (events, people, actions, setting)

2. Entries convincingly record subjective responses to site experience (thoughts, emotions, values, judgments)

3. Entries effectively analyze/ evaluate your experiences (find insights, patterns, meaning, causes, effects)

Well done

Satisfac-tory

Unsatis-factory

Assessment in Sociology and Anthropology

Focus groups of graduating students Given a scenario appropriate to the discipline, a

faculty facilitator asks questions related to outcomes faculty have identified in 3 areas: concepts, theory, methods.

2 faculty observers use 0-3 scale to rate each student on each question

GROUP scores are discussed by all faculty Murphy & Goreham

North Dakota State University

Methods of Assessment Paper and pencil tests Individual or group projects Portfolios Observation of practice Observation of simulated practice Analysis of case studies Attitude or belief inventories Interviews and focus groups Surveys

Assessing Student GrowthThe Portfolio - Some Examples of Content

Course assignmentsResearch papersMaterials from group projectsArtistic productionsSelf-reflective essays (self-assessment)CorrespondenceTaped presentations

Student Electronic Portfolio

Students take responsibility for demonstrating core skills

Unique individual skills and achievements can be emphasized

Multi-media opportunities extend possibilities

Metacognitive thinking is enhanced through reflection on contents

- Sharon J. Hamilton

IUPUI

PRINCIPLES OF UNDERGRADUATE LEARNING (PULs)

1. Core communication and quantitative skills

2. Critical thinking

3. Integration and application of knowledge

4. Intellectual depth, breadth, and adaptiveness

5. Understanding society and culture

6. Values and ethics

Approved by IUPUI Faculty Council May 1998

ePort Goals

Promote clearer understanding among faculty and students of how the curriculum supports increasing mastery of the PULs.

Contribute to assessment of student learning of the PULs at multiple levels of aggregation.

Support student engagement with the PULs over their entire undergraduate experience.

PULs Levels of Competence

Introductory: What all undergraduate students should know and be able to do within the first 26 credit hours.

Intermediate: What all undergraduate students should know and be able to do within the first 56 credit hours.

Advanced: What all baccalaureate recipients should know and be able to do in their major or profession or academic program.

Experiential: Connecting curricular and co-curricular learning

Background Knowledge Probe(Pre-Test – Indirect Measure)

1. ARCHAEOLOGYA. Have never heard of thisB. Have heard of it, but don’t really know what it meansC. Have some idea what it means, but

not too clearD. Have a clear idea what this means

and can explain it- Classroom Assessment Angelo and Cross

Fast Feedback(at end of every class)

Most important thing learned Muddiest point Helpfulness of advance reading

assignments for day’s work in class Suggestions for improving class /

assignmentsBateman & Roberts

Graduate School of Business

University of Chicago

Student Suggestions for Improvement

Install a portable microphone Increase type size on

transparencies Leave lights on when using

projector Don’t cover assigned reading in

detail Provide more examples in class

Outcomes of Classroom Assessment

1. Classes more student centered

2. Focus on learning, not teaching

3. Students take ownership for learning

4. Faculty look for data to make improvements continuously

- Jann Freed

Assessment Update - 1999

In a Comprehensive Assessment Program...

INVOLVE Students Faculty Student Affairs Staff Administrators Graduates Employers

Involve Students1. Set learning expectations in recruiting

2. Communicate learning outcomes in orientation

3. Involve student leaders in promoting learning

4. Involve students in evaluating courses/curricula

5. Let students know their recommendations are used.

Student Advisory Council at Montevallo

A way to provide continuous student assessment

Student Recommendations1 Develop a statement of expected

ethical behaviors for students2 Add a second research course with lab3 Increase comparative psychology4 Add terminals for statistics lab5 Increase opportunities for research,

writing, and speaking

Guidance from Alumni Alumni surveys emphasized that graduates

valued skills in writing, speaking, working collaboratively, and information literacy

Now the Faculty Senate’s General Education Committee has developed 5 learning elements, at least 3 of which must be integrated in any course approved for general education

-Michael Dooris

Penn State University

Junior Course in Professional Communication

Teams devise promotional event for non-profit agency

Campaign theme and rationale Logo and other visual materials Event outline

Faculty, agency representative, peers evaluate the plan

- A. J. Johnson

Alverno College

Assessment in Fine Arts— Portfolio review and/or audition

for every student every semester by

Panel of faculty, students, community representatives, staff or faculty outside fine arts

Results - Students creativity, conceptualization, technique have improved

— Alec Testa— Eastern New Mexico University

Involving EmployersCombination of survey and focus groups

for employers of business graduates Identified skills, knowledge, personality attributes

sought by employers Encouraged faculty to make curriculum changes Motivated student to develop needed skills Strengthened ties among faculty, students, employers

- Kretovics & McCambridge

Colorado State University

Colorado State UniversityCollege of Business

Curriculum changes based on employer suggestions:

1 credit added to Business Communications for team training and more presentations

Ethics & social responsibility now discussed in intro courses

New Intro to Business course emphasizing career decision-making

More teamwork, oral & written communication, problem-solving in Management survey courses

- Kretovics & McCambridge

Authentic AssessmentAt

Southern Illinois University - Edwardsville

Business - Case Study Analysis with Memo Education - Professional Portfolio Psychology - Poster on Research Project Engineering - Senior Design Project Nursing - Plan of Care for Patient

Responses to AssessmentAt

Southern Illinois University - EdwardsvilleBusiness - More case studies and research

Education - More practice in classroom management

Psychology - Curriculum change in statistics Engineering - More practice in writing and

speaking Nursing - Simulation lab with computerized

patients

Virginia Commonwealth University

1. First-year English students wrote 1

response / week for 15 weeks

2. Faculty-student affairs teams read

essays

3. Sample of writers interviewed 18

months later

Responses to findings:

Central advising center, new advising handbook, multicultural workshops

2004-2005

The Year of Accountability

. . . Peter Ewell

Assessment Update

September 2005

National Center for Public Policy in

Higher Education

Measuring Up 2004-5-State Pilot:

•National Adult Literacy Survey•Licensure and grad admissions scores•College Learning Assessment

(or ACT Work Keys)

3 Themes Common to 4 Reports

1. Accountability is now about the outcomes of student learning

2. Accountability is more focused on serving the public interest

3. Accountability now emphasizes public disclosure

. . . Peter Ewell Assessment

Update September 2005

External Accountability Measures

are Summative rather than Formative

and Lead to Compliance

or Creative LOCAL Initiatives

The Future Need for evidence of accountability will

increase More faculty will recognize benefits of

assessment More electronic assessment methods will be

developed More sharing of assessment methods will

take place Faculty will learn more about learning and

student learning will improve