Michelle A. Duda, Ph.D. Melissa Van Dyke, LCSW Chris Borgmeier, Ph.D. Susan Davis, NC SPDG Margaret...

Michelle A. Duda, Ph.D.

Melissa Van Dyke, LCSW

Chris Borgmeier, Ph.D.

Susan Davis, NC SPDG

Margaret McGlinchey, MI SPDG

Evidence-based Professional Development

2011 SPDG Regional Meeting

Feb 2011Washington, D.C.

A Few Questions: What do we mean when we use the term,

“professional development?”

What outcomes are we attempting to achieve?

What do we know about the professional development strategies that are likely to achieve particular outcomes?

How can we monitor and improve the quality of professional development over time?

Overcoming Barriers with

Common language

Common frameworks

Defined approach (best practices)

Common measures (commitment to continuous quality improvement)

Common PD Challenges

COMMON LANGUAGEOvercoming Barriers :

Training

“the process of bringing a person to an agreed standard of proficiency, by practice and instruction”

http://www.thefreedictionary.com/training

“the acquisition of knowledge, skills, and competencies as a result of the teaching of vocational or practical skills and knowledge that relate to specific useful competencies.”

http://en.wikipedia.org/wiki/Training

Adult Learning Defined

“a collection of theories and methods for describing the conditions under which the processes of learning are optimized (Merriam, 2001; Trotter, 2006; Yang, 2003).”

Trivette, Dunst, Hamby, O’Herin, 2009, p. 1

Professional Development

"Professional development ... goes beyond the term 'training' with its implications of learning skills, and encompasses a definition that includes formal and informal means of helping teachers not only learn new skills but also develop new insights into pedagogy and their own practice, and explore new or advanced understandings of content and resources.”

Modified from http://www.ncrel.org/sdrs/areas/issues/educatrs/profdevl/pd2prof.htm

Professional Development (cont.)

“. . . [This] definition of professional development includes support for teachers as they encounter the challenges that come with putting into practice their evolving understandings about the use of new skills and technology.”

Modified from http://www.ncrel.org/sdrs/areas/issues/educatrs/profdevl/pd2prof.htm

Reflection: Language Barriers

How has issues with terminology, definition or language impacted your work?

COMMON FRAMEWORKSOvercoming Barriers :

Effective Interventions

Actual SupportsYears 1-3

Outcomes Years 4-5

Every Teacher Trained

Fewer than 50% of the teachers received some training

Fewer than 10% of the schools used the CSR as intended

Every Teacher Continually Supported

Fewer than 25% of those teachers received support

Vast majority of students did not benefit

Aladjem & Borman, 2006; Vernez, Karam, Mariano, & DeMartini, 2006

Longitudinal Studies of a Variety of Comprehensive School Reforms

Implementation Science

What Works

Implementation Drivers Common features of successful

supports to help make full and effective uses of a wide variety of innovations

Core Implementation

Components

Core Implementation

Components

© Fixsen & Blase, 2008

Improved Outcomes for . . .

Com

pete

ncy

Driv

ers

Com

pete

ncy

Driv

ers O

rganization Drivers

Organization D

rivers

LeadershipLeadership

Program/Initiative (set of practices)

How:

What:

Why:

Capacity to provide direction/vision of process

Staff capacity to support students/families with the selected practices

Institutional capacity to support staff in implementing practices with fidelity

Core Implementation

Components

Core Implementation

Components

© Fixsen & Blase, 2008

Improved Outcomes for . . .

Com

pete

ncy

Driv

ers

Com

pete

ncy

Driv

ers O

rganization Drivers

Organization D

rivers

LeadershipLeadership

Program/Initiative (set of practices)

How:

What:

Why:

Capacity to provide direction/vision of process

Institutional capacity to support staff in implementing practices with fidelity

Professional DevelopmentProfessional Development

© Fixsen & Blase, 2008

Performance Assessment (Fidelity)

Coaching

Training

Selection

Systems Intervention

Facilitative Administration

Decision Support Data System

Integrated & Compensatory

Com

pete

ncy

Driv

ers

Com

pete

ncy

Driv

ers O

rganization Drivers

Organization D

rivers

LeadershipLeadership

Adaptive Technical

Improved Outcomes for . . .

Program/Initiative (set of practices)

Why Focus First on Teachers’ Behavior?

In education, the Teacher IS THE INTERVENTION

Build supports in relationship to what the Teacher needs to be competent

Create the conditions under which teachers can do the right thing for the right reason at the right time to maximize positive results

Wide ranging inputs (individuals with all their past history and current realities)

Build alignment of implementation practices and outcomes on what needs to happen at the point of the “learning exchange”

Improved outcomes for students

Participants with improved knowledge and skills around

identified practice(s)

• Think about those who are the participants in (receivers of) your professional development activities…

• What features would you suggest be in place around the Competency, Organization and Leadership drivers so that professional development is provided in a “host environment” to support sustainability and promote fidelity?

Participants of Professional Development

Participants with improved knowledge and skills around

identified practice(s)

Professional development providers

• Think about those who are the providers of your professional development activities…

• What features would you suggest be in place around the Competency, Organization and Leadership drivers so that your providers conduct quality professional development?

Providers of Professional Development

The NCSIP Process: Focus Of Work

DEFINED APPROACHOvercoming Barriers:

“Best Practices”

The BIG QUESTION:

What outcomes are you hoping to

achieve?

Use training “alone” strategies if…

You want to disseminate new information

You want to create “buy in”

You want to clarify “truths” and dispel “myths”

You are talking with a broad audience

Use training “alone” strategies …

To increase knowledge about the effective program or practice related to. . . underlying theory of change intervention components rationales of key practices

To increase familiarity with the use of new skills

To increase awareness of what it feels like to begin to use new skills and to receive expert feedback

Training “Best Practices”

Theory grounded (adult learning)

Skill-based

Data-based (pre and post testing)

Feedback to Selection and Feed Forward to Supervision

Trainers have been trained and coached

SAMPLE: Reading Foundations Training Pre and Post Test Items

Some tips to consider:

Pre-post test items should be linked to training/teaching objectives

Data from pre-post tests should be used for training revisions, developing coaching service delivery plans, etc.

Presentation of pre-post test should emphasize the non-punitive nature of this type of assessment and that it will be shared back with the participants

Be a good model of using data to get better

Training “Best Practices”

Theory grounded (adult learning)

Skill-based

Feedback to Selection and Feed Forward to Supervision

Data-based (pre and post testing)

Trainers have been trained and coached

What do we know about each

What do we know about each

of these “best practices?”

of these “best practices?”

Adult Learning “Best Practices”

The most effective training includes learner experiences related to planning, application, and deep understanding

Use a diverse array of Adult Learning methods (“where 5 or 6 adult learning method characteristics were used, the average effect size was almost 1.25”)

Learners need to be engaged actively in the learning process

Multiple learning experiences, large doses of learner self-assessment and reflection, instructor facilitated learner assessment

Small numbers of participants, multiple occasions

Trivette, Dunst, Hamby, O’Herin, 2009, p. 10 -11

Adult Learning “Best Practices”

Trivette, Dunst, Hamby, O’Herin found best practices in each of the following categories:

Introducing Information

Illustrate/demonstrate

Practicing

Evaluation

Reflection

MasteryTrivette, Dunst, Hamby, O’Herin, 2009, p. 6 - 8

Skill-based “Best Practices”

Behavior Rehearsals (vs. Role Plays)

Knowledgeable Feedback Providers

Practice to Criteria

Data Based “Best Practices”

Develop and use pre/post tests to determine to what extent knowledge and skill levels are being improved

Outcome data collected and analyzed

Fidelity measures collected and analyzed

Training

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

N = 7 N = 6 N = 3 N = 7 N = 8 N = 9

A B C D E F

Workshops

Pe

rce

nt

of

TI C

om

po

ne

nts

De

mo

ns

tra

ted

PRE

POST

Collins, S. R., Brooks, L.E., Daly, D.L., Fixsen, D.L., Maloney, D.M., & Blase, K. A. (1976)

Staff Training

Feedback “Best Practices”

Feed Forward of pre/post data to Coaches/Supervisors

Feedback of pre/post data to Selection and Recruitment

Feedback “Best Practices”

Of course, there are even more opportunities to benefit from feedback…

© Fixsen & Blase, 2008

Performance Assessment (Fidelity)

Coaching

Training

Selection

Systems Intervention

Facilitative Administration

Decision Support Data System

Integrated & Compensatory

Com

pete

ncy

Driv

ers

Com

pete

ncy

Driv

ers O

rganization Drivers

Organization D

rivers

LeadershipLeadership

Adaptive Technical

Improved Outcomes for training participants

Effective Training Practices

Implementation-informed PD

If Professional Development is in support of a well-defined, effective intervention. . .

. . . then Professional Development includes

Training

Coaching

Performance assessment

and Selection

Training, plus coaching

Some well-defined interventions (EBPs) will require additional support to teachers post-training: Joyce and Showers (2002) data

» With coaching = 95% use in classrooms

» Without = 5% use in classrooms

Rogers, Wellens, & Conner (2002) data

» About 10% of what is taught in business workshops actually is put into practice

Significant data across domains that feedback improves performance

Coaching “Best Practices”

Design a Coaching Service Delivery Plan

Use multiple sources of information for feedback – Direct observation is critical

Provide regular feedback to all “Drivers”

Develop accountability structures for Coaching – Coach the Coaches! Regular review of adherence to Coaching Service Delivery

Plan Multiple sources of information for supervisor feedback

Performance Assessment Best Practices

Transparent Processes – Orientation

What, When, How, Why

Use of Multiple Data Sources

Context

Content

Competency

Tied to positive recognition – not used ‘punitively’

Selection “Best Practices”

Job description clarity about accountability and expectations

Pre-Requisites are related to “new practices” and expectations (e.g. basic group management skills).

Interactive Interview Process:

Behavioral vignettes and Behavior Rehearsals

Assessment of ability to accept feedback

Assessment of ability to change own behavior

EXAMPLE: Qualifications of Potential SIP II Foundation Trainers

Successfully complete the Reading Foundations course including all of the tasks (level 2) and meet the passing criteria (80%) on the Structure of Language Test.

Demonstrate mastery of the content of the Reading Foundations course as well as the ability to deliver this information to others. Mastery of content also includes knowledge of the texts identified as basic to the course.

Provide appropriate review and feedback of tasks completed by participants in the Reading Foundations course.

Enhancing Professional Development

Promotes adult learning and changes adult behavior

Empowers individuals to improve their craft

Ensures that new skills are used and/or current skills are refined

Integrates selection, training, coaching, and performance assessment to promote the development of a highly effective workforce

Supports the creation of a “hospitable environment” to allow for these new behaviors to be used and improved

Core Implementation

Components

Core Implementation

Components

© Fixsen & Blase, 2008

Improved Outcomes for . . .

Com

pete

ncy

Driv

ers

Com

pete

ncy

Driv

ers O

rganization Drivers

Organization D

rivers

LeadershipLeadership

Program/Initiative (set of practices)

How:

What:

Why:

Capacity to provide direction/vision of process

Institutional capacity to support staff in implementing practices with fidelity

Professional DevelopmentProfessional Development

Discussion Question

To what extent are current approaches for

professional development designed to

develop the necessary competence

in participants to skillfully use the

most effective educational

approaches to accomplish the goal of

improved outcomes for students?

COMMON MEASURESOvercoming Barriers :

“Commitment to Continuous Quality Improvement”

Integrating All We Know

A well-defined and effective intervention

Clarity about the “active ingredients”

Clear criteria to assess performance (that correlates with the desired outcomes)

An effective approach to training

An effective coaching system

An effective performance assessment process

Use of selection “best practices”

Improvement Cycles

Improving PD Effectiveness

Collect and use data (pre/post tests to determine to what extent knowledge and skill levels are being improved)

Develop the capacity of the trainers (establish clear performance criteria, based on “what works,” then select, train, coach, and assess trainers to criteria)

Improve the ways in which training participants are “prepared” for training (create readiness)

Questions to Consider

1. What do we mean when we use the term, “professional development?”

2. What outcomes are we attempting to achieve?

3. What do we know about the professional development strategies that are likely to achieve particular outcomes?

4. How can we monitor and improve the quality of professional development over time?

5. What are some resources and tools we can use to support sustainable and aligned professional development?

For More Information

Michelle A. Duda duda@unc.edu

Melissa Van Dykemelissa.vandyke@unc.edu

Chris Borgmeiercborgmei@pdx.edu

For More InformationState Implementation and Scaling up of

Evidence-based Practices (SISEP)

Dean Fixsen, Karen Blase, Rob Horner, George Sugai

www.scalingup.org

Concept paper

Annotated bibliography

Data on scaling up

Scaling up Briefs

www.implementationconference.org

Evidence-based

Fixsen, D. L., Naoom, S. F., Blase, K. A., Friedman, R. M. & Wallace, F. (2005). Implementation Research: A Synthesis of the Literature. Tampa, FL: University of South Florida, Louis de la Parte Florida Mental Health Institute, The National Implementation Research Network (FMHI Publication #231).

Download all or part of the monograph at:

http://www.fpg.unc.edu/~nirn/resources/detail.cfm?resourceID=31  

Implementation Research: A Synthesis of the Literature

Thank You for your Support

Annie E. Casey Foundation (EBPs and cultural competence)

William T. Grant Foundation (implementation literature review)

Substance Abuse and Mental Health Services Administration (implementation strategies grants; national implementation awards)

Centers for Disease Control & Prevention (implementation research)

National Institute of Mental Health (research and training grants)

Juvenile Justice and Delinquency Prevention (program development and evaluation grants

Office of Special Education Programs (scaling up capacity development center)

Administration for Children and Families (child welfare reform; capacity development)

Duke Endowment (child welfare reform)