®®

Are we asking the right questions about professional development on

national surveys?Insights from cognitive interviews

Are we asking the right questions about professional development on

national surveys?Insights from cognitive interviews

Laura Desimone Kerstin Carlson Le Floch

Susie Ansell James TaylorMeisha Fang American Institutes for Research

Vanderbilt University

Laura Desimone Kerstin Carlson Le Floch

Susie Ansell James TaylorMeisha Fang American Institutes for Research

Vanderbilt University

®®

Professional development central to school improvement, yet challenging for surveys

Cognitive interviews is an

effective technique for

identifying specific problems

and possible remedies

Improving Surveys of Professional Development

®®

Cognitive interview data from survey development processes National Longitudinal Study of NCLB, National Longitudinal Evaluation of CSR, Longitudinal Evaluation of the

Effectiveness of School Interventions

Approaches in national surveys NAEP, ECLS, SASS, FRSS, and NELS

Data sources

®®

Cognitive Interviews: Teachers were unsure of what should be considered professional development – faculty meetings, informal meetings, release time?

National Surveys: Great variation in definitions of what respondents should and should not count as professional development

Implications: Need to acknowledge that teachers receive professional development through formal and informal means, surveys should enable teachers to reflect full range of activities, but better specified.

What counts as professional development?

®®

Cognitive Interviews: teachers don’t always follow instructions in stem (!) but tend to think about current year; also didn’t recall all activities.

National Surveys: Some questions were anchored to the current school year, others asked about different

time periods. Implications: Research shows it is helpful to anchor

questions to a particular event, such as the beginning of the school year

Recalling professional development activities

®®

Content vs. Methods

®®

Overlap and Differences by Subject

®®

Impact/ChangeEveryone knows (especially IES)that to attribute cause,Random assignment is the best!

But why can’t we just ask teachersLike we’ve always done before

How well did it work…(on a scale of 1 to 4)?

®®

Content Focus

The Concept of a Conceptual Framework

Active Learning

Collective Participation

Coherence

Duration

Structure/Type

Increase in Teacher Knowledge & Skills

& Change in Practice

®®

More Sophisticated Ideas about Teacher Learning

®®

What are Appropriate Current and Continuing Efforts?

Comparing across surveys Conducting cognitive interviewsUsing a conceptual framework

(coincidentally, exactly what we’re doing!)