HP Technology for Teaching Worldwide Higher Education Conference Portola Plaza HotelMonterey, CA February 3, 2006 Measuring the Impact Daryl E. Chubin,

HP Technology for Teaching Worldwide Higher Education Conference

Portola Plaza Hotel—Monterey, CAFebruary 3, 2006

Measuring the Impact

Daryl E. Chubin, Ph.D.

American Association for the Advancement of Science

www.aaascapacity.org

2006 HP Technology for Teaching Conference

Faculty Hold the Key

“While faculty members may try to do the best they can in class using familiar methods of instruction, they seldom work systematically at improving the methods themselves. Few faculties . . . assess how much their students are learning, identify deficiencies, develop and test possible remedies, and ultimately adopt those approaches that prove most successful.”

Derek Bok, The Chronicle of Higher Education, 12/16/05


On “Learning”

“At bottom, a student’s real opportunity to learn is a function of his or her relation to a teacher in a classroom in a complex but specific learning environment. The best indicator of this opportunity . . . is student learning—the achievement of high levels of skills, knowledge, and understandings in challenging subject areas . . .”

Shirley Malcom, Promises to Keep, Report to the National Education Goals Panel, 11/15/93


To Tell Your Story . . .

Commit to keeping score—and project evaluation Determine what matters and what’s measurable—quantify

when possible Establish a baseline or “pre” condition Focus on “value-added” assessment—difference between

capability on entry and on exit (“post”) Find/adapt/use templates Put numbers into context—qualitative, descriptive, interpretive

Reflect, and learn, from the experience!


Project Evaluation: Analog to Program Evaluation

Chief evaluation prototypes:

Formative: ongoing, process, feedback

Summative: gauging impact, attributing to

intervention, claiming success

See BEST Rating Form for Assessing Education Programs/ Practices,

A Bridge for All (2004), www.bestworkforce.org



Core Data Elements

Class Composition (Learners): The Study Population

Program Character: Student activities

Your behavior

Learning Outcomes & Impacts: Short-term

Longer-term

Your adjustments/modifications


Modes of Measurement

Direct observation

Analysis of documented student performance (assignments, major project, exams)

Comparison of class performance with other course, college, or national disciplinary data

Interviews & focus groups

Consultation with other faculty familiar with the student, the course content, and/or the technology


Guiding Principle

Measure what you value—instead

of valuing what you measure . . .


Other Sources of Inspiration, Practical Advice, & Existence Proofs

National Center for Academic Transformation, “The Roadmap to Redesign” http://www.thencat.org/PlanRes/R2R_PrinCR.htm

Project Kaleidoscope, “What Works Visions”

http://www.pkal.org/collections/WhatWorks.cfm


On “Change”

"People hate change when it is someone else's plan, when it is imposed on them, when they are told what to do and exactly how they must do it, when they are threatened with punishment if they don't do it. People love change when it is shaped by them, when they are in control of it, when it is their chance to make a difference. In fact, then they don't even call it 'change' — it's a project, a venture, a dream come to life. It's their passion turned into a professional pursuit."

Rosabeth Moss Kanter, 2004

May we all catch this spirit!


Breakout Groups will . . .

Engage in table conversations/ brainstorming

Start to fill in the project-specific worksheet

Report out, one per table, illustrative variables that fit one or more project examples

Be better prepared to report/confer with team back home


Then the Conversation Continues . . .

Email me: [email protected]

Call me: 202-326-6785

Consult the web: www.aaascapacity.org

Documents

HP Technology for Teaching Worldwide Higher Education Conference Portola Plaza HotelMonterey, CA February 3, 2006 Measuring the Impact Daryl E. Chubin,