Upload
others
View
1
Download
0
Embed Size (px)
Citation preview
Second Annual Race to the Top
Professional Development Evaluation
Report Part II—Local Outcomes Baseline Study
Authors:
Jenifer Corn, Avril Smart, Lauren Bryant, Jennifer
Maxfield, Meredith Walton, Ruchi Patel, Sara Pilzer Weiss,
Brandy Parker, Shaun Kellogg Friday Institute for
Educational Innovation
Contributors:
Nate Barrett, Julie Marks, Tina Patterson, Carolina Institute
for Public Policy
March 2013
Consortium for Educational Research and Evaluation–North Carolina
Acknowledgements
We wish to thank Lynne Johnson, Maria Petrie-Martin, Neill Kimrey, Yvette Stewart, Eliz
Colbert, Robert Sox, and Stephen Jackson of the North Carolina Department of Public
Instruction for their time and assistance with data collection. We would also like to thank NCDPI
Division staff who led the professional development sessions we observed, as well as the
teachers and staff who participated in our site visits and focus group. We are grateful to Trip
Stallings and Glenn Kleiman from North Carolina State University’s Friday Institute for
Educational Innovation for valuable feedback on drafts of this report, and to Larissa Leroux for
editorial assistance.
RttT PD Evaluation: Year 2, Part II—Local Outcomes Baseline
March 2013
Consortium for Educational Research and Evaluation–North Carolina
Table of Contents
Executive Summary ........................................................................................................................ 3
Introduction ..................................................................................................................................... 7
Purpose of the RttT Evaluation and of this Report ...................................................................... 8
Overview of RttT Professional Development Activities and Supports Addressed in this Report .... 9
2011-12 NCDPI-RESA Sessions ........................................................................................... 10
2012 Summer Institutes .......................................................................................................... 10
Method .......................................................................................................................................... 12
Statewide Data Sources ............................................................................................................. 13
LEA Professional Development Coordinator Survey ............................................................ 13
RttT Omnibus Survey ................................................................................................................ 13
Longitudinal Sample.................................................................................................................. 14
School Sample Selection Process ........................................................................................... 14
Description of the Sample ...................................................................................................... 15
Longitudinal Data Sources and Procedure ................................................................................ 15
CLASS Observation Protocol ................................................................................................ 16
Teacher Professional Development Survey ........................................................................... 17
LEA Professional Development Coordinator Interviews....................................................... 19
Principal Interviews................................................................................................................ 20
Teacher Focus Groups ............................................................................................................ 20
Limitations ................................................................................................................................. 20
Results ........................................................................................................................................... 22
Evaluation Question 3.A.: To what extent did leader participants improve practices for support
of organizational change, including capacity to provide high-quality local professional
development aligned to RttT priorities? .................................................................................... 22
Planning, Accessing, and Implementing Local Professional Development........................... 23
Alignment with RttT Priorities ............................................................................................... 27
Quality of Local Professional Development .......................................................................... 29
Summary of Findings for Evaluation Question 3A ................................................................ 31
Evaluation Question 3.B.: How did school culture/organizational conditions change to support
RttT priorities? ........................................................................................................................... 32
Changes in Conditions related to Transition to the New State Standards .............................. 32
Changes in Conditions related to Data-based Decision Making and Revised Teacher
Evaluation Process ................................................................................................................. 34
RttT PD Evaluation: Year 2, Part II—Local Outcomes Baseline
March 2013
Consortium for Educational Research and Evaluation–North Carolina 2
Evaluation Question 3.C.: To what extent did teachers improve classroom practice? ............. 36
Next Steps for the Professional Development Evaluation ............................................................ 42
Continued Evaluation of the RttT Annual Professional Development Cycle ........................... 42
Continued Evaluation of the Distinguished Leadership in Practice (DLP) Principal Institutes ...... 43
Continued Collaboration with NCDPI Staff .............................................................................. 43
References ..................................................................................................................................... 44
Appendix A. RttT Professional Development Teacher Survey Items .......................................... 45
Appendix B. LEA Professional Development Coordinator Interview Protocol ........................... 50
Appendix C: Principal Interviews Interview Protocol .................................................................. 52
Appendix D. Teacher Focus Group Protocol ................................................................................ 54
RttT PD Evaluation: Year 2, Part II—Local Outcomes Baseline
March 2013
Consortium for Educational Research and Evaluation–North Carolina 3
SECOND ANNUAL RACE TO THE TOP PROFESSIONAL DEVELOPMENT
EVALUATION REPORT, PART II—LOCAL OUTCOMES BASELINE STUDY
Executive Summary
The North Carolina Race to the Top (RttT) professional development plan is an expansive and
multi-faceted effort to increase student achievement by updating the knowledge and skills of the
state’s entire K-12 public education workforce. This initiative is driven by a host of recent
changes, including: adoption of new Common Core State Standards and North Carolina Essential
Standards; increased use of data to inform classroom and school decisions; rapid changes in the
technologies and digital resources available for teaching and learning; new teacher and
administrator evaluation processes; and an increased emphasis on formative assessment to
inform instructional decisions.
The human resources challenge of the initiative—to provide the state’s 100,000 teachers and
2,400 principals with professional development that will enable them to extend their knowledge,
improve professional practices, and, ultimately, increase student achievement overall and close
achievement gaps among student groups—is formidable. The timeframe (the four-year period of
the grant), diversity of the State (from large metropolitan local education agencies [LEAs] to
small, rural, and resource-limited LEAs, many of which continue to struggle under the weight of
fiscal constraints), and expectations (to create a statewide professional development
infrastructure that can be sustained after RttT funding ends) only increase that challenge. The
RttT professional development evaluation is being conducted in full recognition of these
circumstances, as well as of the deep commitment of the members of the RttT Professional
Development Implementation Team. The intent of the evaluation is to provide data-driven
information that can support reflection about and improvement of this effort.
Four general questions guide the evaluation:
1. State Strategies: To what extent did the state implement and support proposed RttT
professional development efforts?
2. Short-Term Outcomes: What were direct outcomes of State-level RttT professional
development efforts?
3. Intermediate Outcomes: To what extent did RttT professional development efforts
successfully update the NC education workforce?
4. Impacts on Student Performance: To what extent are gains in student performance
outcomes associated with RttT professional development?
The Evaluation Team is providing this second annual assessment of progress in three separate
but related reports. This report—the Local Outcomes Baseline Study— provides a baseline (first-
year) assessment of the State’s progress toward updating the education workforce in North
Carolina (Evaluation Question 3). For this report, the Evaluation Team (a) collected and
analyzed relevant data from all 115 LEAs, and (b) identified a purposeful sample of 27 schools
to examine more deeply the extent to which LEA and school staff increased capacity to provide
high-quality professional development. The other two reports address (1) the state’s progress in
RttT PD Evaluation: Year 2, Part II—Local Outcomes Baseline
March 2013
Consortium for Educational Research and Evaluation–North Carolina 4
delivering face-to-face professional development statewide, and (2) implementation and impact
of the State’s Online Professional Development.
Summary of Major Baseline Findings
To address Evaluation Question 3, the Team examined the extent to which participation in the
state’s professional development efforts impacted (a) local capacity to provide high-quality
professional development, (b) shifts in local organizational conditions to support RttT priorities,
and (c) changes to instructional practice. Results from these baseline analyses will be used in
future reports to gauge progress toward updating the education workforce statewide.
Evaluation Question 3.A.: To what extent did leader participants improve practices for support
of organizational change, including capacity to provide high-quality local PD aligned to RttT
priorities?
Planning and Accessing Professional Development.
For most schools, focus group and interview data reveal that school-level professional
development was informed by what was being offered at the state level and typically was
designed around resources available at the LEA level. In line with the state-endorsed “train the
trainer” model, LEA-level Professional Development Coordinators indicated that they were
primarily responsible for identifying and disseminating the most relevant professional
development resources and making them available to principals and teachers in their LEAs. Most
LEA Coordinators mentioned using tools like wikis, website postings, and staff development
newsletters and calendars to condense professional development resources and make them more
easily accessible to teachers and principals. However, despite the multiple methods of
dissemination described by LEA leaders, data indicate that some principals and teachers still
remained uninformed about NCDPI-support professional development opportunities.
Implementing Professional Development.
Most Professional Development Coordinators agreed that their LEA leadership had the capacity
to implement and plan professional development, but fewer believed that their LEA leaders used
data to inform their professional development decisions. Some LEAs called on teams of school-
level leaders to develop implementation strategies. In most schools, professional development
training was implemented through Professional Learning Communities (PLCs), and in some
LEAs, training opportunities were extended beyond the school year.
Alignment with RttT Priorities.
LEA Professional Development Coordinators statewide and teachers in the representative school
sample held different perceptions about whether professional development helped teachers build
their knowledge and skills related to some RttT priorities, such as understanding revised state
standards (91% of Coordinators but only 64% of teachers agreed that it helped) and deepening
their content knowledge (81% and 58%, respectively). For other priority areas, the perceptions of
both groups were more closely aligned, but were typically low (e.g., only 51% and 52%,
respectively, agreed that teachers built knowledge and skills related to revised state assessments).
RttT PD Evaluation: Year 2, Part II—Local Outcomes Baseline
March 2013
Consortium for Educational Research and Evaluation–North Carolina 5
In addition, while 92% of Coordinators reported that LEA-provided RttT professional
development was aligned with and built upon existing professional development initiatives, only
64% of the coordinators agreed that their LEAs utilized data on staff technology proficiency
when planning RttT professional development. Furthermore, only 60% agreed that their LEAs
have plans for how to use the NCDPI Professional Development Leads in each region for
support in their delivery of professional development. Overall, these differing perceptions
suggest a lack of consistency among LEA PD coordinators in their efforts to align professional
development with RttT priorities at the local level.
Quality of Local Professional Development
As they reflected on the academic year, about 70% of teachers surveyed agreed or strongly
agreed that their local professional development experiences were both sustained and coherent,
though fewer believed that they had time to reflect on and experiment with what they learned—a
sentiment that was echoed in focus group responses. Teachers generally gave lower ratings for
their experiences with LEA-provided professional development than did LEA Professional
Development Coordinators, but interview and focus group data highlighted principals’ and
teachers’ positive experiences with local coaches and specialists, as well as their positive
impressions of the value of PLCs.
Evaluation Question 3.B.: How did school culture/organizational conditions change to
support RttT priorities?
Changes in Conditions related to Transition to the New Standards
Most LEA Professional Development Coordinators agreed that their LEAs have strategies in
place for communicating about availability of, collaborating on, and integrating state resources
into professional development related to revised state standards, but preparation for integrating
other aspects of the curriculum with those standards varies. Results from the RttT Omnibus
Survey showed that nearly all teachers felt strongly about their own preparation for the transition
to Common Core but were less sure about whether school leaders or other teachers in their
school really understood how to implement the new standards effectively.
Changes in Conditions related to Data-based Decision Making and the Revised Teacher
Evaluation Process
Between 2010-11 and 2011-12, there were slight decreases in teacher perceptions of their schools’
use of data for decision-making, as well as in their perceptions of the new teacher evaluation
process, but both changes may be the result of a sizeable increase in response rates. The Evaluation
Team will monitor annual survey results to determine whether these patterns persist.
Evaluation Question 3.C.: To what extent did teachers improve classroom practice?
Data gathered for this section provide baseline information about current learning activities for
comparison to similar data in subsequent years of RttT. Overall, Math teachers most frequently
reported daily instructional time on instructional strategies related to new standards, followed by
English Language Arts (ELA), Science, and Social Studies teachers. Across each content area:
RttT PD Evaluation: Year 2, Part II—Local Outcomes Baseline
March 2013
Consortium for Educational Research and Evaluation–North Carolina 6
ELA: The highest percentage of teachers estimated that students spent time almost daily on listening skills (76%) and on general vocabulary (66%).
Math: The highest percentage of teachers reported that on almost a daily basis, students made sense of problems (80%), persevered in solving problems (72%), and used appropriate tools
strategically (67%).
Science: Teachers reported that their students practiced participating in hands-on activities (32%), completed activities with a real-world context (24%), and used tools to gather data
(19%).
Social Studies: Students spent time almost daily on recognizing and appreciating contributions of diverse cultural groups (30%), demonstrating chronological thinking (29%),
and analyzing cause-and-effect relationships (28%).
On average, observed classrooms in the sample schools received midrange ratings across the five
dimensions of the Evaluation Team’s standard classroom observation protocol1 that are most
directly aligned with implementation of the new Common Core and Essential Standards (Regard
for Student Perspectives, Quality of Feedback, Instruction Dialogue, Analysis and Problem
Solving, and Content Understanding). Across content areas, teachers sometimes scaffolded
discussion in the classroom and made attempts to integrate student feedback. ELA classrooms
were scored slightly higher on average than were other core courses in the frequency and quality
of examples of three of the dimensions of effective teaching (Regard for Student Perspectives,
Quality of Feedback, and Instruction Dialogue). Social Studies classrooms received the lowest
average ratings for observed frequency and quality of the other two dimensions (Analysis and
Problem Solving and Content Understanding). Finally, teacher surveys indicated that while some
formative assessment practices were common across classrooms, others were employed
infrequently. These observation data mirror results from teachers’ self-report surveys about
frequency of effective, standards-based teaching practices.
Summary of Results
Overall, the baseline results presented in this study indicate that schools and LEAs have taken
some initial steps toward developing a process for professional development that supports RttT
priorities, including transition to the new state standards, implementation of formative and
summative assessments, use of data to support instruction, and effective utilization of the new
North Carolina Educator Evaluation System. These aggregated results provide a cursory baseline
assessment of Year 1 progress toward achieving and sustaining local capacity to provide high-
quality professional development, supporting shifts in local organizational conditions to support
RttT priorities, and encouraging changes in instructional practice.
1 Classroom Assessment Scoring System, or CLASS (http://www.teachstone.org/about-the-class/). In all, there are
12 CLASS domains.
http://www.teachstone.org/about-the-class/
RttT PD Evaluation: Year 2, Part II—Local Outcomes Baseline
March 2013
Consortium for Educational Research and Evaluation–North Carolina 7
Introduction
Providing high-quality, accessible professional development to all teachers and principals is a
key component of the professional development plan funded by North Carolina’s federal Race to
the Top (RttT) grant. The United States Department of Education’s grant application process
recognized the important role that professional development must play in the successful
implementation of the RttT education reforms by requiring states to develop comprehensive
strategies for both the expansion of their professional development offerings and the evaluation
of the effectiveness of that professional development. North Carolina’s RttT-funded professional
development plans are ambitious, with a top-level goal of updating the entire education
workforce to ensure that each of the state’s 100,000 teachers and 2,400 principals has the
knowledge and skills necessary to foster student achievement.
The challenges addressed by the North Carolina Department of Public Instruction’s (NCDPI)
RttT Professional Development Implementation Team are formidable. The Implementation
Team must develop strategies that, in coordination with Local Education Agencies (LEAs),
Regional Education Service Agencies (RESAs), and other organizations, will engage all of the
state’s teachers and administrators in professional development that will extend their knowledge
and guide them in making significant changes in their professional practices in order to increase
overall student achievement and close achievement gaps among student groups. The plan
requires differentiation of professional development to meet the needs of educators with a wide
range of backgrounds, preparation, experience, and expertise, ranging from the over 10,000
experienced National Board Certified teachers in North Carolina to the over 20,000 teachers who
are in their first three years of teaching and who in many cases entered teaching through an
alternative licensure pathway. The plan requires reaching teachers from kindergarten through
high school, across all disciplinary areas. The plan also requires professional development for
principals, assistant principals, curriculum specialists, and all of the other administrators
involved in guiding and supporting teachers through transitions to new standards, student
assessments, data systems, technologies, teacher evaluations, and overall expectations for both
themselves and their students. All of this is to be accomplished within the four-year period of the
grant, across a large and diverse state with many small, rural, and resource-limited LEAs. In
addition, the plan is expected to result in a statewide professional development infrastructure that
can be sustained after RttT funding ends. While the grant provides significant funding, the
professional development initiative must proceed during a time of severe economic constraints
that limit additional state and local resources that can be brought to bear on the challenges; this
limitation is far greater than was anticipated when the RttT proposal was developed.
The RttT professional development evaluation is being conducted with full recognition of the
enormous challenges being addressed by the Professional Development Implementation Team
and the deep commitment of the members of the team to do so. The Evaluation Team further
recognizes that the RttT professional development plan is breaking new ground: There are no
established, proven models for creating a professional development system of this scale to
support so many educators through so many changes in so short a period of time. The Evaluation
Team’s intent for this report is to provide data-driven information that can support reflection
about and improvement of the RttT professional development effort.
RttT PD Evaluation: Year 2, Part II—Local Outcomes Baseline
March 2013
Consortium for Educational Research and Evaluation–North Carolina 8
Purpose of the RttT Evaluation and of this Report
The roles of the RttT Evaluation Team are to (1) document the activities of the RttT initiatives;
(2) provide timely, formative data, analyses, and recommendations to help the initiative teams
improve their ongoing work; and (3) provide summative evaluation results in Year 4 of the
project to determine whether the RttT initiatives met their goals and to inform future policy and
program decisions to sustain, modify, or discontinue initiatives after the grant-funded period.
The RttT Evaluation Team is documenting the professional development initiative activities and
collecting data about participation in, satisfaction with, and the impact of locally- and state-
supported professional development activities. In addition, the team is collecting information
through LEA and school field visits and teacher and administrator surveys to inform summative
evaluation analyses that examine the extent to which participation in RttT professional
development activities has increased the capacity of the education workforce to deliver effective
instruction. Ultimately, the evaluation will use state assessment data to gauge changes in student
performance. The full plan is described in greater detail in Appendix A.
Four general questions guide the overall evaluation effort, with a number of more specific
questions embedded in each one. These overall questions are organized to reflect the general
sequence in which they can be addressed over the four years of RttT:
1. State Strategies: To what extent did the state implement and support proposed RttT
professional development efforts?
2. Short-Term Outcomes: What were direct outcomes of state-level RttT professional
development efforts?
3. Intermediate Outcomes: To what extent did RttT professional development efforts
successfully update the North Carolina education workforce?
4. Impacts on Student Performance: To what extent are gains in student performance
outcomes associated with RttT professional development?
The Year 2 professional development evaluation serves three primary purposes, each of which is
addressed in a separate report, as described below. Each of the three reports is organized around
specific evaluation questions; this document is Report II of the Year 2 RttT evaluation.
Report I: RttT Statewide Face-to-Face Professional Development Formative Evaluation—Report I documents the current status and direct outcomes of the state’s RttT
face-to-face professional development efforts in the NCDPI Annual Professional
Development Cycle described below, addressing questions under Overall Evaluation
Question 1 (State Strategies) and questions under Overall Evaluation Question 2 (Short-Term
Outcomes). It is intended to contribute to the RttT Professional Development Implementation
Team’s ongoing efforts to provide the most supportive and useful professional development
possible for teachers and administrators.
Report II: RttT Professional Development Local Outcomes Baseline Study—Report II (this report) provides detailed information concerning near- and longer-term impacts of
RttT PD Evaluation: Year 2, Part II—Local Outcomes Baseline
March 2013
Consortium for Educational Research and Evaluation–North Carolina 9
statewide face-to-face RttT professional development efforts at the local level. The
Evaluation Team collected and analyzed relevant data from all 115 LEAs and identified a
purposeful sample of 27 schools to examine more deeply the extent to which LEA and
school staff increased capacity to provide high-quality professional development and focused
on shifts in local organizational conditions to support RttT priorities, and changes to
instructional practice. That is, this section provides baseline data for Evaluation Question 3
(Intermediate Outcomes) and related sub-questions:
A. To what extent did leader participants improve practices for support of organizational change, including capacity to provide high-quality local PD aligned to RttT priorities?
B. How did school culture/organizational conditions change to support RttT priorities?
C. To what extent did teachers improve classroom practice?
Report III: RttT Online Professional Development Evaluation Report—Report III also addresses Evaluation Questions 1, 2, and 3 (State Strategies, Short-Term Outcomes, and
Intermediate Outcomes), but with a focus on all RttT-supported online professional
development. It provides detailed information about the extent to which the online
professional development components of the RttT proposal have been carried out, and the
immediate impact of RttT online professional development efforts on educators.
The next major assessment, scheduled for delivery in Fall 2013, will consider short-term
outcomes in more detail and will begin to address intermediate outcomes, while reports in
following years (after the local professional development programs are in operation) will address
the impact on teachers and, to the extent possible, students. Since the schedule of the Evaluation
Team’s deliverables calls only for annual reports, the team will actively coordinate with NCDPI
to share preliminary survey data reports with appropriate staff, in order to provide timely
information as the Professional Development Implementation Team continues to work with
LEAs and charter schools and develops additional resources.
Overview of RttT Professional Development Activities and Supports Addressed in this Report
As shown in the diagram of the Annual Professional Development Cycle provided by NCDPI
(Figure 1, following page), the overall professional development plan is built around annual
cycles comprised of Summer Institutes, formative support for LEA and charter school
Professional Development Leadership Teams, and additional face-to-face support sessions
provided by the NCDPI RttT Professional Development Implementation Team in collaboration
with RESAs. Through implementation of this Annual Professional Development Cycle, NCDPI
aims to both guide and support the development of local professional development through a
high-quality, systemic, blended approach to effective professional development, defined as job-
embedded, research-driven, data-informed, professional, community-based, and aligned to the
RttT initiatives. A blended approach to professional development is designed to address the
needs of LEAs, charter schools, and individual educators through face‐to‐face sessions and online resources, such as learning modules or webinars.
RttT PD Evaluation: Year 2, Part II—Local Outcomes Baseline
March 2013
Consortium for Educational Research and Evaluation–North Carolina 10
Figure 1. NCDPI Annual Professional Development Cycle
2011-12 NCDPI-RESA Sessions
NCDPI offered over 100 face-to-face NCDPI-RESA partnership sessions across the eight North
Carolina State Board of Education regions from August 2011 to June 2012. Sessions lasted
anywhere from three hours to two days and focused on Content Support (Common Core and
Essential Standards), Fidelity Support, Teacher Effectiveness/New Accountability Model, North
Carolina Professional Training Standards for Principals and Assistant Principals, Detailed
Scope of Work/Technical Assistance, Principal Training for Common Core and Essential
Standards, and Principal and Assistant Principal Training (ITES Standards). These sessions were
developed and facilitated by the NCDPI RttT Professional Development Implementation Team in
partnership with RESA staff.2
2012 Summer Institutes
Six two-day Summer Institutes were held across the state to provide opportunities for local
professional development leaders to expand their learning about the Common Core State
Standards and the North Carolina Essential Standards. LEA and charter school teams, comprised
of up to 18 members, including key leaders from each LEA and charter school, attended these
summer trainings. These sessions were conducted in collaboration with LEAs and facilitated by
2 For descriptions of these sessions, please see Table 10 in Report I (Report I: RttT Statewide Face-to-Face
Professional Development Formative Evaluation).
RttT PD Evaluation: Year 2, Part II—Local Outcomes Baseline
March 2013
Consortium for Educational Research and Evaluation–North Carolina 11
the NCDPI RttT Professional Development Implementation Team. This year’s Summer Institutes
focused on how to facilitate learning for all students and included sessions organized by topic:
Content Strands, Leaders with Leaders, Facilitative Team Time, and Assessment and
Accountability.
RttT PD Evaluation: Year 2, Part II—Local Outcomes Baseline
March 2013
Consortium for Educational Research and Evaluation–North Carolina 12
Method
In order to examine more deeply the extent to which LEA and school staff increased their
capacity to provide high-quality professional development at the local level, the Evaluation
Team collected and analyzed relevant data from all 115 LEAs (state strategies), and identified a
purposeful sample of 27 schools (short-term outcomes). Eight data sources were used over the
course of the 2011-12 academic year and the summer of 2012 (Table 1). The Evaluation Team
used a mixed methods approach to address the broad evaluation questions discussed above.
Quantitative methods included surveys and structured observations. Qualitative approaches
included interviews and focus groups, as well as open-ended survey questions. This section
discusses each of the data sources used and the corresponding methodology. Percentages
presented in the following tables have been rounded to the nearest whole number; therefore, not
all percentages add up to exactly 100.
Table 1. RttT Evaluation Data Sources, 2011-12
*RttT priorities include 1) transition to new state standards, 2) implementation of new formative/summative
assessments, 3) use of data literacy for instruction, 4) use of technology, and 5) effective use of the revised North
Carolina Educator Evaluation System.
Data source Purpose
Statewide
RttT LEA Professional
Development Coordinator
Survey
Gather information about local capacity for providing high-
quality professional development
RttT Omnibus Teacher
Survey
Assess change across a wide range of constructs that may be
influenced by the collective set of NC RttT activities
Longitudinal
CLASS Observation Protocol Gather information about classroom practices
RttT Professional
Development Teacher
Survey
Gather information about teachers’ attitudes, experiences,
knowledge, etc., about professional development and RttT
priorities*
RttT Professional
Development LEA
Coordinator Interviews
Discuss local capacity for providing high-quality professional
development
RttT Professional
Development Principal
Interviews
Discuss principals’ experience and recommendations about
RttT professional development related to RttT priorities*
RttT Professional
Development Teacher Focus
Groups
Discuss teachers’ experience and recommendations about RttT
professional development related to RttT priorities*
RttT PD Evaluation: Year 2, Part II—Local Outcomes Baseline
March 2013
Consortium for Educational Research and Evaluation–North Carolina 13
Statewide Data Sources
LEA Professional Development Coordinator Survey
The Evaluation Team created and distributed a survey to LEA Professional Development
Coordinators to gain information on how the LEAs were developing and implementing
professional development programs that addressed the RttT initiatives, as well as whether they
were accessing and using NCDPI’s RttT resources and events. The survey items asked the
participants to use a 7-point Likert scale to rate their LEA in terms of how well their local
professional development efforts helped their teachers move toward RttT initiative goals (e.g.,
“Professional development experiences in my district this year have helped teachers implement
the state Standard Course of Study”). Participants also were asked to indicate the progress of
their LEA in building the capacity to continue serving their educators after the completion of the
RttT grant (e.g., “In my district, leaders have the skills to plan and design high-quality
professional development”).
The survey was distributed via email in February 2012 to LEA leadership with a request that it
be forwarded to the Professional Development Coordinator. Due to this deployment method, the
data were cleaned to remove any non-Professional Development Coordinators who may have
taken the survey. The Team received a total of 274 responses to the survey. After restricting the
data to those who specifically identified themselves as serving in a professional development
coordinator capacity, the final sample size used in the analysis was 159. Survey responses were
received from all eight regions, from 92 of the 115 LEAs (Table 2).
Table 2. LEA Professional Development Coordinator Survey Response Rate by Region, 2012
Region
Number of LEAs that
Completed the Survey
Number of LEAs in
Region Response Rate
1 13 15 87%
2 13 14 93%
3 12 14 86%
4 10 12 83%
5 10 15 67%
6 7 9 78%
7 16 19 84%
8 11 17 65%
Total 92 115 80%
RttT Omnibus Survey
The overall research plan for the four years of the NC RttT evaluation includes annual
administration of an Omnibus Teacher and Principal Survey to a statewide sample in February of
each school year. The survey was designed to assess change across a wide range of constructs
that may be influenced by the collective set of NC RttT activities, with items in several
constructs corresponding to specific initiatives, including professional development.
RttT PD Evaluation: Year 2, Part II—Local Outcomes Baseline
March 2013
Consortium for Educational Research and Evaluation–North Carolina 14
The Omnibus Survey is comprised of 170 items across 23 dimensions. Examples of key
dimensions include teacher-leadership respect, teacher-teacher trust, teacher knowledge sharing,
and teacher-student relationships. Each respondent received a random sub-sample of the
questions to decrease respondent burden. The survey was administered online via Qualtrics, and
the sample of questions for each participant was estimated to take between 15 and 30 minutes to
complete.
For this report, five dimensions on the survey were identified to help understand the evolution of
the professional development component of RttT. The tables in this report were generated using
data from individual teacher responses to items in the following dimensions:
1. Quality of Professional Development
2. Alignment of Professional Development
3. Data-Driven Instruction
4. Evaluation of Teachers
5. Formative Assessment
For the first four of these dimensions, items were measured on a 7-point Likert scale: (1)
Strongly disagree, (2) Disagree, (3) Slightly disagree, (4) Neither agree nor disagree, (5) Slightly
agree, (6) Agree, and (7) Strongly agree. Items in the fifth dimension, Formative Assessment,
asked how often different formative assessment strategies were used; frequency was measured
on a 5-point scale: (1) Never, (2) A few times a year, (3) Once or twice a month, (4) Once or
twice a week, and (5) Almost daily. Since the nature of the questions and scale of the data were
different, the results for the Formative Assessment dimension are reported separately, after the
results for the first four professional development dimensions.
Longitudinal Sample
School Sample Selection Process
In order to gain a more accurate perspective on the impacts of the RttT professional development
initiatives on educators across the state, a purposeful sample of schools was selected to
participate in the longitudinal descriptive study. School type, professional development ratings,
and student achievement were the three primary variables that influenced the selection of the
schools. The Evaluation Team categorized all schools in the state by type (elementary, middle, or
high school), tercile of recent professional development quality (high, midrange, or low) based
on the results from the 2010 TWC survey, and tercile of school performance composite scores
(high, midrange, or low) based on the North Carolina ABCs of Public Education school
accountability model. School type, urbanicity, achievement status, teacher characteristics,
student characteristics, and other variables also were considered in the selection of schools for
the sample. Finally, considerations were made to ensure that the schools were geographically
representative. This selection process yielded a stratified sample of 27 schools.
RttT PD Evaluation: Year 2, Part II—Local Outcomes Baseline
March 2013
Consortium for Educational Research and Evaluation–North Carolina 15
Description of the Sample
This section describes the sample schools in general terms; specific information that could be
used to identify individual schools was removed to prevent violation of confidentiality.
Schools. The longitudinal sample included nine traditional elementary schools (grades K-5),
seven middle schools (grades 6-8), nine high schools (grades 9-12), and two schools that blend
either middle and elementary grades or middle and high school grades. Across all 27 schools in
our sample, 83% of their Adequate Yearly Progress goals in the 2010-11 academic year were
met, and collectively, 75% of their students were at or above grade level. Approximately 19% of
schools were located in cities, 11% were suburban, 26% were in a town, and 44% were rural.
Schools were distributed across all eight State Board of Education regions.
Figure 2. Geographic Distribution of Longitudinal Professional Development Study Schools
Teachers and students. Across the sample schools, about 26% of teachers had advanced degrees,
18% were National Board Certified, and 96% were fully licensed. Eighteen percent of teachers in
the sample had three or fewer years teaching experience.
The ethnicities of students in the sample were comparable to those across the state (North
Carolina Department of Public Instruction, 2012). On average, about 55% were White/
Caucasian, 26% were Black/African American, 14% identified as Hispanic/Latino, 4% were
multiracial, 1% were Asian, and less than 1% were American Indian/Native American or Pacific
Islander. Approximately 55% of the students in the sample were eligible to receive free or
reduced-price lunch.
Longitudinal Data Sources and Procedure
During the 2011-12 academic year, the Evaluation Team gathered quantitative and qualitative
data on professional development activities in the 27 schools. Quantitative data were gathered
through structured classroom observations and teacher surveys. Qualitative data included semi-
structured interviews and focus groups with LEA Professional Development Coordinators,
principals, and teachers in the sample schools. All interviews and focus groups were facilitated
by Evaluation Team members. The following section describes each data source and the
procedures used to analyze the data collected.
RttT PD Evaluation: Year 2, Part II—Local Outcomes Baseline
March 2013
Consortium for Educational Research and Evaluation–North Carolina 16
CLASS Observation Protocol
The Classroom Assessment Scoring System (CLASS) was used to gather data regarding
teachers’ progress in the transition to new state content standards (Common Core and Essential
Standards), which was a primary goal of the RttT-funded professional development. Research
has shown CLASS to be both valid and reliable (Center for Advanced Study of Teaching and
Learning, 2011), and it can be used in a wide range of classroom situations. In addition, the
tool’s depth offers several advantages over similar tools. CLASS offers different versions for
multiple grade levels, and its 7-point rating scale offers more rating flexibility and refinement
than do scales found in other tools considered by the Evaluation Team. All RttT evaluators who
conducted the classroom observations completed a certification process that consisted of two
days of training and successful completion of a CLASS Reliability Test.
CLASS was developed to assess classroom quality based on observed instructional interactions
between teacher and students, as well as productivity evident in classroom settings (Teachstone,
2012). The system has a total of 12 dimensions, but for this evaluation, the Team chose to focus
on the 5 CLASS dimensions that most closely align with instructional strategies outlined in the
Common Core and Essential Standards (Regard for Student Perspectives, Content
Understanding, Analysis and Problem Solving, Quality Feedback, and Instructional Dialogue;
Table 3). Each dimension was scored using the 7-point scale, with 1 being the lowest.
During the 2011-12 academic year, the Evaluation Team observed over 120 hours of teaching in
core content classrooms (English-Language Arts [ELA], Math, Social Studies, and Science)
within our 27 sample schools. Observers spent between one and two hours in each classroom in
both the fall and spring semesters.
Table 3. CLASS Observation Tool Dimensions Chosen for Use in the RttT Evaluation
CLASS Dimension Description
Regard for
Student
Perspectives
(RSP)
This dimension focuses on the extent to which the teacher is able to meet and capitalize on
the social and developmental needs and goals of adolescents by providing opportunities for
student autonomy and leadership. Also considered are the extent to which student ideas and
opinions are valued and content is made useful and relevant to adolescents.
Content
Understanding
(CU)
Content understanding refers to both the depth of lesson content and the approaches used to
help students comprehend the framework, key ideas, and procedures in an academic
discipline. At a high level, this refers to interactions among the teacher and students that lead
to an integrated understanding of facts, skills, concepts, and principles.
Analysis and
Problem Solving
(APS)
This dimension assesses the degree to which the teacher facilitates students’ use of higher
level thinking skills, such as analysis, problem solving, reasoning, and creation through the
application of knowledge and skills. Opportunities for demonstrating metacognition (i.e.,
thinking about thinking) are also included.
Quality of
Feedback (QF)
Quality of feedback assesses the degree to which feedback expands and extends learning and
understanding and encourages student participation. In secondary classrooms, significant
feedback may also be provided by peers. Regardless of the source, the focus here should be
on the nature of the feedback provided and the extent to which it “pushes” learning.
Instructional
Dialogue (ID)
Instructional dialogue captures the purposeful use of dialogue—structured, cumulative
questioning and discussion that guide and prompt students—to facilitate students’
understanding of content. The extent to which these dialogues are distributed across all
students in the class and across the class period is important to this rating.
RttT PD Evaluation: Year 2, Part II—Local Outcomes Baseline
March 2013
Consortium for Educational Research and Evaluation–North Carolina 17
CLASS data were compiled for all classroom observation sessions during the 2011-12 academic
year, for a total of 208 observed classrooms (grades 1 through 12) and 702 observations in all.
The data were grouped by school performance level and by professional development rating (as
described above). Summary statistics for the various groupings were then calculated for the
classroom characteristics and the CLASS variables of interest listed above.
On average, classrooms contained one adult and 20 students. ELA represented over 30% of the
content observed in classrooms, followed by Math and Science, each of which represented 25%
of observed content. Twenty-one percent of classrooms taught Social Studies, and the remaining
focused on non-core content. Generally, classrooms observed for this sample used “whole class”
instructional formats for some or all of the instruction observed (69% of classrooms), followed
by “small group” and “individual instruction” formats (approximately 40% each). It is important
to note that because the study included elementary schools, Evaluation Team members had the
option to indicate observation of multiple content areas for each classroom characteristic per
observation. For example, in an elementary classroom, an observer may see both ELA and Math
taught, with the teacher using multiple instructional formats for each, all within the same
observation period.
Teacher Professional Development Survey
The Teacher Professional Development Survey (Appendix A) was designed to gauge teachers’
perceptions of the quality and utility of state- and locally-provided professional development
training. More specifically, items on the survey asked teachers to rate LEA leadership capacity
for providing professional development, the potential impacts of professional development on
teachers’ skills and knowledge in the classroom, and the integration of Common Core and
Essential Standards into classroom instructional practices. The items about instructional practices
aligned to the new standards were developed in close collaboration with NCDPI leaders in the
Curriculum and Instruction division for the core content areas.
Principals in each sample school disseminated the survey to teachers in their schools between
October 2011 and March 2012. Survey items included both Likert agreement scales and
frequency scales. Of the 1,211 participants who started the survey, 22 participants indicated that
they either did not teach a class (e.g., some participants indicated that they were counselors or
principals), were not certified to teach (e.g., teacher aides), or left the “certified to teach”
question blank; these participants were removed from the initial sample, reducing the sample to
1,189.
In general, teachers who completed the Teacher Professional Development Survey were
representative of teachers across the state. Similar to state-reported ratios in the Public Schools
of North Carolina statistical profile for the 2011-12 school year3 teachers in this sample were
75% female (n = 891), and over 80% were Caucasian/White (n = 890; Table 4, following page).
3 North Carolina Public Schools Statistical Profile, 2011–12
http://apps.schools.nc.gov/pls/apex/f?p=1:21:0::NO:::
RttT PD Evaluation: Year 2, Part II—Local Outcomes Baseline
March 2013
Consortium for Educational Research and Evaluation–North Carolina 18
Table 4. Participant Representation by Ethnicity, Teacher Professional Development Survey
Ethnicity
Percentage of
Respondents
American Indian/Alaska Native 1%
Asian*
RttT PD Evaluation: Year 2, Part II—Local Outcomes Baseline
March 2013
Consortium for Educational Research and Evaluation–North Carolina 19
Table 6. Participants’ Education Experience, Teacher Professional Development Survey
Education Experience
Percentage of
Respondents
Bachelor’s degree 34%
Bachelor’s degree plus credits 23%
Certificate of Advanced Study 1%
Master’s degree 25%
Master’s degree plus credits 12%
Doctorate* 0%
I am a lateral entry teacher 4%
*Representation for those holding a doctorate is 0.5%
Response rate. Based on self-reported data from principals in our sample regarding the total
number of teachers in their schools, the overall survey response rate on the Teacher Professional
Development Survey was 77% (Table 7).
Table 7. Survey Response Rates, Teacher Professional Development Survey
School Level
Mean Response
Rate
Elementary School 83%
Middle School 56%
High School 87%
Overall 77%
Response rates for each question on the survey varied for a few reasons. First, participants were
able to skip questions. Second, many items offered the response option of “Not Applicable”; this
response option was treated as missing when calculating proportions of responses. Finally, the
response options for the Quality of Professional Development and the District Capacity items
initially contained an error; the “Neither disagree nor agree” option offered only “Agree.”
Because it is not possible to know if the participants who saw this item were influenced by this
erroneous response option, all participants who viewed the incorrect scale were removed from
that section of the analysis.
Analysis of the participant survey data focused on frequency of responses. Data presented from
this survey in the following sections either show proportions of respondents who agreed or
strongly agreed with each item, or they show the frequency of all participant responses,
depending on the scale used.
LEA Professional Development Coordinator Interviews
Principals at each sample school were asked to identify the person in their LEA who coordinated
professional development training and assistance for their school. The Professional Development
RttT PD Evaluation: Year 2, Part II—Local Outcomes Baseline
March 2013
Consortium for Educational Research and Evaluation–North Carolina 20
Coordinators were asked to participate in a 30-minute interview to discuss their LEAs’ capacity
to provide high-quality professional development. The interview protocol (Appendix B)
contained questions specific to strategies used in selecting appropriate professional development
for teachers and staff, integration of RttT priority areas into professional development training,
and the use of state-provided resources. Interviews were conducted between October 2011 and
March 2012.
Principal Interviews
Principals from each sample school participated in two 30-minute semi-structured interviews in
the fall and spring semesters, during which they discussed their experience with professional
development planning and implementation. The interview protocol (Appendix C) contained
questions specific to strategies used in selecting appropriate professional development for
teachers and staff, integration of RttT priority areas into professional development training, and
the use of state-provided resources.
Teacher Focus Groups
Six to eight core content teachers (in English, Math, Science, and Social Studies) from each
school were selected to participate in a 45- to 60-minute focus group. Teachers from each school
were assembled onsite and engaged in a discussion facilitated by an Evaluation Team member.
The focus group protocol (Appendix D) was designed to gather teachers’ opinions of the
availability and use of professional development resources at their school, as well as of their
overall impressions of the transition to the new Common Core and Essential Standards.
The Evaluation Team collected over 45 hours of interview and focus group data. After each
audio recording was transcribed, Atlas.ti software was used to facilitate qualitative analysis of
the data. An a priori coding scheme was developed based on the evaluation questions outlined in
the introduction of this report. The coding scheme was refined and expanded as the team
interacted with the data. Results from the analysis contributed to the baseline outcomes in this
report.
Limitations
Due to changing schedules and available resources on the RttT Professional Development
Evaluation Team, a few data collection strategies did not reach their full potential.
CLASS Observation Protocol: Evaluators were able to observe only a small proportion of the teachers in the schools in our longitudinal study, and only for a short time—four teachers in a
school for two hours at a time, twice a year. The intent is not to rate teachers but to use this
valid and reliable classroom observation tool to systematically record observed instructional
interactions between teacher and students. For the purposes of this study, these data will be
used to describe teachers’ transition to the new state content standards (Common Core and
Essential Standards), as evidenced by effective instructional practices required for the
teaching to the new standards.
Teacher Professional Development Survey: When transferring the survey to the online survey portal, a clerical error resulted in two “agree” response options for the items about Quality of
RttT PD Evaluation: Year 2, Part II—Local Outcomes Baseline
March 2013
Consortium for Educational Research and Evaluation–North Carolina 21
Professional Development and LEA Capacity. Because it is not possible to know if
participants were influenced by this erroneous response option, all participants who viewed
the incorrect scale for those constructs were removed from analysis.
Specific strategies have been identified to address these limitations for the remainder of the
evaluation and are outlined in the final section of this report, Next Steps for the Professional
Development Evaluation.
RttT PD Evaluation: Year 2, Part II—Local Outcomes Baseline
March 2013
Consortium for Educational Research and Evaluation–North Carolina 22
Results
This report provides a baseline (first-year) assessment of the State’s progress toward updating
the education workforce in North Carolina, as measured by the status of intermediate outcomes
related to that goal (Figure 3). These outcomes are expected to result from educator participation
in statewide face-to-face RttT professional development events designed to impact (a) local
capacity to provide high-quality professional development, (b) shifts in local organizational
conditions to support RttT priorities, and (c) changes to instructional practice. That is, this report
provides baseline data for addressing Evaluation Question 3— To what extent did RttT
professional development efforts successfully update the North Carolina education workforce?—
and related sub-questions.
Figure 3. Logic Model of Local-Level Outcomes for Statewide RttT Professional Development
*RttT priorities for North Carolina include: successfully transitioning to the new standards, implementation of
formative and summative assessments, use of data to support instruction, and effective utilization of the new North
Carolina Educator Evaluation System (NCEES).
Evaluation Question 3.A.: To what extent did leader participants improve practices for support
of organizational change, including capacity to provide high-quality local professional
development aligned to RttT priorities?
In order to assess progress toward meeting intermediate outcomes related to updating the North
Carolina education workforce at this point in the RttT grant, qualitative and quantitative data
were collected to: (1) describe whether local capacity has been built to plan, access, and
implement high-quality professional development (Figure 4, following page); (2) provide
baseline information about the extent to which there have been organizational shifts at the school
and LEA level related to the RttT priorities; and (3) provide baseline information about the
quality of the professional development being provided.
RttT PD Evaluation: Year 2, Part II—Local Outcomes Baseline
March 2013
Consortium for Educational Research and Evaluation–North Carolina 23
Figure 4. Logic Model of Outcomes for Statewide RttT Professional Development—Local
Capacity Focus
*RttT priorities for North Carolina include: successfully transitioning to the new standards, implementation of
formative and summative assessments, use of data to support instruction, and effective utilization of the new North
Carolina Educator Evaluation System (NCEES).
Planning, Accessing, and Implementing Local Professional Development
Planning and Accessing Local Professional Development. Focus group and interview data
revealed that, for most schools, school-level professional development was informed by what
was being offered at the state level and typically was designed around resources available at the
LEA level. When asked about professional development training around specific RttT priority
areas in particular, some coordinators and principals noted a strong reliance on state and LEA
guidance (e.g.: “We go with whatever is provided by our LEA;” “I think our professional
development is through what is offered by the state”). In the few cases where local-level
leadership were more directly involved in the planning process, Professional Development
Coordinators and principals highlighted cross-LEA sessions where professional development
strategies were exchanged among LEAs in their region. One Coordinator’s comments
summarized the appreciation expressed by local leadership for the chance to engage in the
planning process:
When you’re working within a district, you’re isolated within that district and you don’t
know what else is going on around you. It was good to hear what smaller districts could
do, and what some of the larger districts comparable to [our] county could be doing.
Since we are all going toward the new assessments and new curriculum, it’s good to hear
RttT PD Evaluation: Year 2, Part II—Local Outcomes Baseline
March 2013
Consortium for Educational Research and Evaluation–North Carolina 24
what others are doing to prepare their teachers. Then we can decide within our county
cohort what things we think might work, if we need to go back in and redesign our plan,
or move forward with what we think our best practice is.
Some schools also relied on in-house experts or outside vendors (e.g., Reading 3D or PD 360) to
plan supplemental professional development training for their teachers.
Qualitative data from the sample schools detailed the ways in which information about
professional development availability typically was disseminated by LEA leadership to teachers
and principals at the school level. In line with the state-endorsed “train the trainer” model, LEA-
level Professional Development Coordinators indicated that they were primarily responsible for
identifying and disseminating the most relevant professional development resources and making
them available to principals and teachers in their LEAs. Most LEA Coordinators mentioned
using tools like wikis, website postings, and staff development newsletters and calendars to
condense professional development resources and make them more easily accessible to teachers
and principals. Other methods of increasing awareness of professional development
opportunities included teachers and administrators sharing information at faculty meetings and in
subject-area or grade-level Professional Learning Communities (PLCs).
Despite the multiple methods of information dissemination described by LEA leaders, data
revealed that some principals and teachers still remained uninformed about NCDPI-supported
professional development opportunities. For example, when focus group moderators asked
teachers whether they were aware of any online professional development opportunities offered
by NCDPI, teachers from four schools were unaware of any of the online modules being offered,
and teachers from four other schools indicated that they were aware of or had accessed the NC
FALCON modules developed prior to RttT but were not aware of the new online modules
released in the summer and fall of 2011. This lack of awareness was likely due to either a LEA
Professional Development Coordinator’s decision to postpone the rollout of the modules,
unresolved decisions about what modules or module components to use, or, as in the case of two
LEAs, a decision not to make the modules mandatory because of duplication of prior local
efforts or concerns over their quality.
While most schools were able to locate and access some professional development resources,
more evidence is needed to fully assess changes in local professional development training
patterns. While there are indications that, at the state level, NCDPI leadership has recognized the
need to create a better structure to describe how to connect state-implemented professional
development to professional development training at the LEA and school levels, a need for
specific communication strategies for professional development at the local level persists, as
evidenced by teachers’ and principals’ lack of awareness of professional development resources,
despite LEA Professional Development Coordinators’ express role in disseminating state-
provided professional development resources. Differences in professional development needs
across schools and the availability of resources impact North Carolina educators’ ability to
translate state-level professional development resources into local-level training. This baseline
data will be used as a point of comparison for future evaluations of LEA and school ability to
locate and access professional development resources.
RttT PD Evaluation: Year 2, Part II—Local Outcomes Baseline
March 2013
Consortium for Educational Research and Evaluation–North Carolina 25
Implementing Local Professional Development. The Evaluation Team’s approach to examining
LEA and school preparedness to implement professional development was two-fold. First, data
from the statewide LEA RttT Professional Development Coordinator Survey was used to
examine the extent to which LEA leaders believed they had the capacity to implement high-
quality professional development. Second, qualitative data from principals and teachers in our
sample schools provided detailed information about the process by which LEA and school
leadership selected, planned, and implemented professional development at the local level.
Table 8 presents results from the RttT LEA Professional Development Coordinator Survey on
perceptions of LEA leadership capacity to develop professional development programs that suit
the needs of their schools and LEAs.
Table 8. Perceptions of LEA PD Coordinators about LEA Capacity to Implement Professional
Development
In my district, leaders…
Percentage of
Respondents
who Strongly
Agree/Agree
(n = 144-150)
Have knowledge about high-quality professional development defined by research
and national and state standards 83%
Have the skills to plan and design high-quality professional development 83%
Have the skills to implement high-quality professional development 87%
Have the skills to select high-quality professional development 92%
Have positive attitudes about high-quality professional development 94%
Have a district-wide commitment to high-quality professional development 91%
Support Communities of Practice around high-quality professional development 81%
Provide opportunities for networking and support (both online and offline) in high-
quality professional development 77%
Have a district-level strategic plan for professional development in place 86%
Align school-level professional development plans to district plans 80%
Distribute responsibilities for leadership for professional development among
multiple school and district leaders 86%
Use data from teacher performance evaluations to create individual professional
development plans for teachers 68%
Aggregate data from teacher performance evaluations and use data to identify
school/district professional development needs (Teacher Professional Development
Survey: Use data from teacher evaluations to identify school/district professional
development needs)
64%
Use survey data to select, plan, and design professional development 78%
Use summative student data to select, plan, and design professional development 79%
Use formative student data to select, plan, and design professional development 67%
Source: RttT LEA Professional Development Coordinator Survey
RttT PD Evaluation: Year 2, Part II—Local Outcomes Baseline
March 2013
Consortium for Educational Research and Evaluation–North Carolina 26
Survey results revealed that statewide, a large majority of Professional Development
Coordinators agreed or strongly agreed that their LEA leadership had the capacity to implement
and plan various aspects of professional development. For instance, 91% of Professional
Development Coordinators agreed or strongly agreed that their LEA leaders had a commitment
to implementing high-quality professional development, and 86% of Professional Development
Coordinators believed that their LEAs had a strategic plan in place for professional development.
Results were slightly less favorable regarding LEA leaders’ use of data to inform professional
development decisions, with 79% of coordinators agreeing that LEA leaders used summative
student data to select, plan, and design professional development, 67% agreeing that leaders used
formative student data for the same purpose, and only 64% agreeing that LEA leaders used data
from teacher performance evaluation results to identify professional development needs.
The design for professional development training implementation sometimes was a group effort
at the LEA level (for example, collaboration among principals from across an LEA), as
explained by this focus group participant:
This particular year, how we are rolling out the Common Core, we brought in a group of
principals, elementary level, middle school level, and high school level, to look at how
they thought we should roll this out, and who should be trained, and who should be the
ones taking the information back to the schools. We are very large. And so there’s no way
that we’re going to have the face-to-face contact with each teacher within the district. So
we had to develop a plan to use the trainer model, so to speak, in order to at least get the
information out, so that they can begin the teaching of the Common Core.
At the school level, professional development training was designed primarily for delivery
within PLCs or staff meetings. Some schools provided supplemental training resources through
online tools (e.g., Live Binder or Moodle) as additional support for teachers in their transition to
the new state standards. Generally, teachers felt that PLCs were designed to provide an
opportunity to gather with other teachers in their content areas or grade levels to review
professional development resources and walk through any changes in curriculum related to the
new state standards that would affect their instructional practice. For many teachers, PLCs or
staff meetings were their first entrée into professional development training focused on RttT. The
following comments reflect teacher and principal feedback regarding the usefulness of designing
PLCs around professional development goals:
I think obviously the professional learning communities, it’s really helped us to grow
those and it has also enabled us to really utilize that blended approach where we can use
some online, some face-to-face, and we’re using Moodle.
We’re asking teachers to use these [additional] tools to learn the standards themselves.
We’re using the online environment to actually use these modules that DPI set up and
then take them a step further and use the Moodle environment to hold the PLC
conversations. So once they view the module, then we have them go and do online
discussions within their PLC, and so each PLC is doing something a little differently.
Implementation was not limited to 2011-12 school year professional development. Many LEAs
provided summer pre-training opportunities designed to give teachers early chances to explore
RttT PD Evaluation: Year 2, Part II—Local Outcomes Baseline
March 2013
Consortium for Educational Research and Evaluation–North Carolina 27
new Common Core materials and find ways to apply them to their specific grade levels. Most
teachers in the focus groups noted that their experience in summer training was helpful in
preparing them for the use of state-implemented strategies related to Common Core. For a small
proportion of LEAs and schools, implementation of professional development around the new
state standards began one or two years before the 2011-12 school year. One Professional
Development Coordinator commended her LEA for extending professional development
strategies around the Common Core and Essential Standards from the previous year:
Our county is trying to be proactive and they made sure at the end of the last school year,
the 2011-11 school year, that we had the actual Common Core curriculum in our hands,
and then over the summer and through this year, they’re making sure we’re unpacking
the standards; looking at them, understanding for each grade level: Do you know what is
being transitioned out of your grade? Do you know what’s being transitioned into your
grade? I know that there are a lot of counties that are surrounding us that are not doing
those things to prepare. We’re already beginning to teach those things and starting to
close the gap this year so that it won’t be as hard of a transition for the teachers next
year.
Alignment with RttT Priorities
Items from the LEA Professional Development Coordinator Survey and Professional
Development Teacher Survey specifically asked about teachers’ participation in professional
development aligned to RttT priorities (Table 9). Only about half of LEA Professional
Development Coordinators from across the state and teachers in the sample schools agreed or
strongly agreed that teachers built their knowledge and skills for revised state assessments (51%
and 52%, respectively) and teaching diverse student populations (53% and 59%, respectively).
Perceptions of LEA Professional Development Coordinators statewide and teachers in our
representative sample differed in other areas, most particularly about whether teachers built their
knowledge and skills for revised state standards (91% and 64%, respectively) and whether they
deepened their content knowledge (81% and 58%, respectively).
Table 9. Perceptions of Participation in Professional Development on RttT Priorities
To what extent do you feel that teacher knowledge and
skills have been enhanced in each of the following areas as a
result of participation in professional development?
Percentage of Respondents who
Strongly Agree/Agree
PD Coordinators
(n = 146-148)
Teachers
(n = 864-890)
Revised state standards 91% 64%
Instructional materials 73% 58%
Approaches to formative assessment 79% 64%
Use of technology 67% 70%
Strategies for teaching diverse student populations 53% 59%
Deepening content knowledge 81% 58%
Leadership development 71% 53%
Revised state assessments 51% 52%
Revised NC Teacher Evaluation Process 81% 63%
Sources: RttT LEA Professional Development Coordinator Survey, RttT Professional Development Teacher Survey
RttT PD Evaluation: Year 2, Part II—Local Outcomes Baseline
March 2013
Consortium for Educational Research and Evaluation–North Carolina 28
When asked to rate their level of agreement with statements about the alignment of their LEA’s
professional development training and RttT priorities, most coordinators (>85%) agreed that their
LEAs had plans, strategies, resources, and systems in place to support RttT Professional
Development (Table 10). Fully 92% of coordinators reported that LEA-provided RttT professional
development was aligned with and built upon existing professional development initiatives.
Table 10. Professional Development Coordinators’ Perceptions of LEA Capacity to Support RttT
Professional Development
To what extent do you agree/disagree with the following?
Percentage of
Respondents who
Strongly Agree/Agree
(n = 139-144)
Our Race to the Top professional development is aligned with and builds on existing
professional development initiatives in the district. 92%
Our district has . . .
a strategic plan for the deployment of Race to the Top professional development. 92%
an implementation timeline in place for Race to the Top professional development. 90%
a process in place for how we will communicate Race to the Top professional development
initiatives and opportunities to all district staff. 92%
ensured that Race to the Top professional development activities are supported with up-to-
date board-approved policies and procedures. 85%
designated key personnel who are responsible for various components of our Race to the Top
professional development plan. 93%
a system for monitoring and evaluating our Race to the Top professional development plan. 79% communicated with our local testing coordinator to ensure all staff has NC Education login
capabilities. 88%
a comprehensive plan to coordinate the integration of face-to-face and online professional
development. 78%
utilized data on staff’s technology proficiency when planning Race to the Top professional
development. 64%
ensured all staff has equitable access to technology for accessing Race to the Top
professional development resources. 85%
determined that all online resources related to Race to the Top are accessible in the LEA. 89%
specific strategies for collaboration as we roll out Race to the Top professional development. 91%
provided educators an online space (e.g., wiki, website, Moodle, etc.) for sharing resources,
experiences, and/or information to related to RttT. 77%
collaborated with district technology staff to determine the extent of online communication
and collaboration tools available for PD. 72%
designated key personnel to be responsible for monitoring and recording the progress of staff
in completing online and face-to-face Race to the Top PD. 81%
developed a plan for implementing PLCs (online and/or face-to-face) related to Race to the Top. 73% designated key personnel responsible for scheduling and monitoring PLCs related to Race to
the Top. 70%
a plan for how to use the DPI PD Leads in each region for support in our Race to the Top
professional development. 60%
a long-range plan in place for Race to the Top professional development for the next three years. 71% Our district’s Race to the Top professional development . . .
goals are aligned with state initiatives. 94%
team can articulate their role in our Race to the Top professional development plan. 85%
differentiates to meet participants’ needs with additional resources and training. 75%
Source: RttT LEA Professional Development Coordinator Survey
RttT PD Evaluation: Year 2, Part II—Local Outcomes Baseline
March 2013
Consortium for Educational Research and Evaluation–North Carolina 29
However, coordinators highlighted two areas in which they thought LEAs were less effective in
implementing RttT professional development. Only 64% of coordinators agreed that their LEAs
utilized data about staff’s technology proficiency when planning RttT professional development.
In addition, only 60% agreed that their LEAs have plans for how to use the NCDPI Professional
Development Leads in each region for support in their RttT professional development. Similar to
some of the results reported above regarding perceptions of LEA capacity to provide appropriate
professional development, these results also suggest a lack of capacity to use data to guide the
selection, design, and implementation of high-quality professional development.
Quality of Local Professional Development
Data from the RttT Omnibus Teacher Survey provides insight into North Carolina teachers’
general perceptions of the overall quality of professional development they experienced in 2011
and 2012 (Table 11). About 70% of teachers agreed or strongly agreed that professional
development experiences were both sustained and coherent in the 2010-11 and 2011-12 school
years. However, fewer (63%) agreed or strongly agreed that they had enough time to think about,
try, and evaluate new ideas in 2010-11, and that number fell further (to only 56%) in 2011-12.
Table 11. Teacher Perceptions of Overall Quality of Professional Development
Overall, my professional development experiences this year have...
2011
Strongly
Agree/Agree
(n = 1,709)
2012
Strongly
Agree/Agree
(n = 10,011)
Been sustained, rather than short-term. 70% 70%
Been coherently focused, rather than unrelated. 70% 69%
Included enough time to think carefully about, try, and evaluate new ideas. 63% 56%
Source: RttT Omnibus Teacher Survey
Both the RttT Omnibus Teacher Survey and LEA Professional Development Survey results
demonstrate that, although a majority of respondents agreed that their experiences were positive,
teachers generally gave lower ratings for their experiences with LEA-provided professional
development than did LEA Professional Development Coordinators (Tables 11 and 12).
In their assessment of the quality of LEA-provided professional development, 93% of the
Professional Development Coordinators agreed that teachers were provided opportunities to
work productively with colleagues in their school, compared to only 70% of teachers. Similarly,
66% of Coordinators but only 46% of teachers agreed that teachers were provided the
opportunity to work with teachers from other schools, and 81% of Coordinators but only 63% of
teaches thought most of what teachers learned in professional development addressed the needs
of the students in the classroom (Table 12, following page).
RttT PD Evaluation: Year 2, Part II—Local Outcomes Baseline
March 2013
Consortium for Educational Research and Evaluation–North Carolina 30
Table 12. Professional Development Coordinators’ Perceptions of LEA-Level Professional
Development Alignment
To what degree do you agree with the following statements about
the PD experiences provided in your district this year?
Strongly
Agree/Agree
(n = 152-154)a
Strongly
Agree/Agree
(n = 10,011)b
Teachers are provided opportunities to work productively with
colleagues in their school 93% 70%
Teachers are provided opportunities to work productively with
teachers from other schools 66% 46%
Most of what teachers learn in professional development addresses the
needs of the students in their classroom 81% 63%
Sources: a RttT LEA Professional Development Coordinator Survey;
b RttT Omnibus Teacher Survey
Data gathered in interviews and focus groups that was related to the quality of local professional
development was somewhat more encouraging, highlighting principals