“My classroom is a much more positive place”

Incredible Years Teacher programmes —

NZCER Evaluation Summary ReportReport to the Ministry of Education

Cathy Wylie and Rachel FelgateNZCER

ISBN: 978-0-478-16808-2 (print)

ISBN: 978-0-478-16809-9 (web)

RMR-1080

© Ministry of Education, New Zealand 2016

Research reports are available on the Ministry of Education’s website Education Counts:www.educationcounts.govt.nz/publications.

Opinions expressed in this report are those of the authors and do not necessarily coincide with those of the Ministry of Education.

Incredible Years Teacher programmes 3

Evaluation summary

“My classroom is a much more positive place”

Incredible Years Teacher programmes —

NZCER Evaluation Summary Report

Cathy Wylie and Rachel Felgate

4 Incredible Years Teacher programmes

Evaluation summary

The Incredible Years Teacher programmes The Incredible Years Teacher (IYT) is a research-based teacher development programme that is part of the suite of linked Incredible Years (IY) programmes for teachers, parents and children developed by Carolyn Webster-Stratton and her team in the United States. The goal of the interlinked Incredible Years programmes is “to prevent and treat young children’s behaviour problems and promote their social, emotional, and academic competence”.1 Incredible Years Teacher (IYT) is delivered by a pair of IYT-trained group leaders through full-day workshops for groups of around 16–20 teachers, once a month over six months. The group leaders also provide coaching for individual teachers in between workshops. The coaching involves discussion and classroom observation and feedback. Each workshop’s content is detailed in the IYT manual. This manual also provides process guidelines for facilitators, and checklists for planning and review. The programme is intended to be tailored to the specific needs of each participating group of teachers.

IYT is part of the Ministry of Education’s Positive Behaviour for Learning (PB4L) strategy. It was first offered in 2011. It is provided by Resource Teachers of Learning and Behaviour (RTLB), non-governmental organisations (NGOs), such as kindergarten associations, and Ministry of Education staff. More than 7,000 early childhood education (ECE) and primary teachers in New Zealand have now completed IYT.

The evaluation The evaluation focused on the short-term outcomes for teachers and students, the factors enabling positive change, and whether the IYT programme in New Zealand was being delivered as intended. It used material from group leaders and participants in IYT programmes that ran over the first eight months of 2014. We based our data collection and analysis on the theory of change (see p.10) developed with the Ministry of Education and IYT practice leaders. Table 1 shows the data we have used.

Table 1 Data sources for each focus of the IYT evaluation Outcomes for students 428 teacher responses to a NZCER paper survey early in their IYT programme and an

email survey in late 2014 about the ‘target’ child they focused on during the programme

1,103 teacher responses to a NZCER paper survey at their final workshop on their students generally

Outcomes for teachers 1,103 teacher responses to the NZCER paper surveys completed at their first and final IYT workshops

IYT surveys for these 1,103 teachers at the start and end of the IYT programme

239 teachers’ responses to a follow-up email survey in May 2015 re sustainability of learning and practice 8-9 months after the IYT programme

Delivery of IYT in New Zealand

97 IYT group leaders’ responses to an email survey in mid-2014

1,103 teachers’ responses to the paper survey at their final workshop

Factors supporting positive change

All the data sources above

Analysis of responses showed that they were generally representative of the known characteristics of IYT group leaders and participants in New Zealand, so we are confident that the evaluation findings are sound.

1 www.incredibleyears.com

Incredible Years Teacher programmes 5

Evaluation summary

Key finding“My classroom is a much more positive place, and there is so much more time for learning.”

(Participating primary teacher, May 2015)

Outcomes for studentsAt the end of their programme, the majority of teachers reported gains for their students that they attributed to their IYT learning. These gains were that ...

there was less disruptive behaviour in their class or group: reported by 90 percent of the ECE teachers and 75 percent of the primary teachers

students were more focused on their learning work: reported by 85 percent ECE and 70 percent of primary teachers

students had better self-regulation: 82 percent ECE, 75 percent primary

students showed more problem-solving skills: 78 percent ECE 69 percent primary

students could ignore negative behaviours of others more: 74 percent ECE, 74 percent primary.

Comments made by teachers about the IYT programme as a whole were generally very positive, with illustrations given of gains made for student behaviour and attention to learning in their classes overall.

The majority of the ‘target’ students who were a particular focus for teachers in their IYT work were also reported to have made gains. Target students were those whose behaviour was challenging for their teacher. Working individually with their IYT group leader, teachers devised a behaviour plan for their particular target student using the IYT strategies. During the six months of the IYT programme the teacher and group leader then generally discussed the target student’s progress together in relation to the behaviour plan and altered it if needed.

At the end of their IYT programme, 85 percent of the ECE teachers and 78 percent of the primary teachers noted improvements in their target student’s behaviour. Some months later, at the end of 2014, 66 percent of ECE teachers and 47 percent of primary teachers reported that this improvement represented a large positive difference in target students’ behaviour, while almost all the rest felt that small positive differences had occurred.

Marked increases in positive interactions with their peers and teachers, self-management and attention to learning were reported for the target students over the course of 2014. For example:

57 percent of target ECE students often or always persisted with solving a problem at the end of 2014 compared with 21 percent at the start of the IYT programme

45 percent of target primary students often or always followed all class routines without having to be reminded at the end of 2014, compared with nine percent at the start of the IYT programme.

Target student achievement also increased. By the end of 2014, 55 percent of the primary target students were described by their teachers as achieving at a medium or high level for their class compared with 35 percent at the start of 2014.

Parental support for ECE target students also improved.

6 Incredible Years Teacher programmes

Evaluation summary

The gains they saw with their target students encouraged teachers to develop and use behaviour plans with other students in their class. By the end of their IYT programme, 81 percent of ECE teachers and 80 percent of primary teachers reported improvement in the behaviour of these other students also.

Outcomes for teachers Teachers felt their confidence in managing student behaviour increased over the course of the IYT programme.

At the start of the programme: 46 percent of ECE and 55 percent of primary teachers rated themselves as confident or very confident that they could manage their current students’ behaviour.

At the end of the programme: 88 percent of ECE and 74 percent of primary teachers now considered themselves as confident or very confident that they could manage their current students’ behaviour.

Confidence levels in managing student behaviour were maintained 8–9 months after the end of teachers’ IYT programmes (May 2015).

At the end of their programme, 90 percent or more of the ECE teachers and 85 percent or more of the primary teachers thought that what they had gained from IYT included:

more awareness of the value of being proactive in relation to children’s behaviour a much deeper understanding of how to teach social and emotional skills useful strategies to work with children in ways that encouraged their class engagement.

Teachers made real changes in their practice that meant they were more consciously managing behaviour, defining and recognising positive behaviour from students and building children’s social and emotional skills, rather than acting reactively after challenging behaviour and paying counter-productive attention to misbehaviour. There was a marked increase, for example, in positive coaching, problem-solving and anger management strategies, and having a clear classroom discipline plan.

Teachers made more use of IYT strategies by the end of the programme, and they also became more effective in the way they used these strategies.

A marked theme in comments made by the majority of teachers at the end of their programme was that IYT had renewed their sense of agency. IYT gave them the understanding and tools they needed to reframe how their classes operated, with testimonies of new calm for both students and teachers.

Teachers’ comments about the value of their IYT learning remained very positive when surveyed 8–9 months later, in 2015. There were some changes in the frequency of their use of IYT strategies. Half of the IYT teaching techniques we asked about continued to be used by primary teachers often or very often and ECE teachers continued to use a third of the IYT techniques often or very often.

Some IYT teaching techniques appeared to be easier to sustain than others. By May 2015, frequency of use of some of the teaching techniques had dropped, particularly those that involved coaching, modelling and some of the deeper aspects of the IYT approach. Declines in use were most apparent for ECE teachers. It was also noted that there was evidence of an increased use of some of the negative strategies for managing student behaviour among primary teachers.

Incredible Years Teacher programmes 7

Evaluation summary

Delivery of IYT in New Zealand IYT was chosen as a professional development programme to support positive behaviour in New Zealand because of both its content and coherence. Evaluations overseas had shown that it was effective. It provides a structured programme, using video vignettes to illustrate strategies in use by teachers with small groups of students, but also emphasises ‘flexible implementation’ based on participants’ needs, which is consistent with research about effective teacher learning. This programme is not one that can be taken off the shelf and used by any professional development provider. It can only be provided by trained group leaders, who review their own work using IYT tools, and are given feedback from peer coaches who are further along the IYT group leader pathway. Our evaluation therefore looked at the support for group leaders, as well group leader and teacher practice in relation to IYT delivery and learning, as reported in the various surveys used for the evaluation.

Teachers were largely positive about their experiences in the IYT workshops. The IYT group leaders were reported as modelling positive interaction, reflection and inquiry. The workshops engaged teachers and underlined their sense of agency about student behaviour by providing variation in activity, and ensuring that teachers shared their experiences with one another, so that the strategies were able to be seen as ‘live’, and not as something outside their own scope. This interaction with other teachers was highly valued. Teachers’ views about the IYT video vignettes however suggest that they are not a straightforward resource for supporting change. On the one hand, they were valued as demonstrations and for stimulating discussions; on the other hand, they sometimes also caused scepticism given the differences in group size and adult resources between the American and New Zealand contexts.

Some variability around what IYT group leaders were doing in their work with individual teachers between workshops was evident. A substantial minority of teachers did not ‘usually’ discuss their target child’s progress or the self-report inventory with their group leader. Teachers who usually had such discussions reported more gains in practice, and therefore it is important that these regular discussions occur for all teachers undertaking the IYT programme.

In mid-2014, IYT group leaders themselves were not all getting the ongoing support envisaged in the IYT model. The time available for IYT work was less than needed for 62 percent of the group leaders, particularly time to work individually with teachers between workshops.

Almost half the group leaders were not able to keep improving their practice through coaching with accredited peer coaches. Such access was associated with group leaders’ use of other reflective processes in the IYT model that support ongoing fidelity, and confidence with presenting the IYT approach, such as the use of checklists in video review. Those who were most recently trained had less of this kind of support to ensure fidelity and grow them as group leaders. There were also some issues evident around ensuring that newer group leaders had sufficient experience of leading groups to keep developing, and that effective practice (positive framings of the video vignettes where teachers react negatively to them, for example) can be shared. Responsibility for these aspects fell to different organisations, making it hard at times to ensure coherence.

Since the evaluation survey data were collected, the Ministry of Education has undertaken IYT workforce planning, and has recently contracted Massey and Canterbury Universities to provide group leader development and support. It will be important for the ongoing quality of IYT delivery in New Zealand that the issues identified in this section can be addressed.

8 Incredible Years Teacher programmes

Evaluation summary

Factors supporting positive change Gains from IYT and positive views of it were across the board: the programme appears to be effective regardless of teaching experience or ethnicity.

IYT programme delivery and the teachers’ own contexts both played a part in changes made by teachers and gains for students.

IYT programme delivery The quality of the IYT group leaders — their teaching, preparation and interest in teachers (eg, by modelling what teachers can do in their own classrooms) — matters. It also matters that they work well with individual teachers between workshops, to both share their knowledge about strategy choice and use, and model ongoing review and inquiry.

The behaviour plan work for ‘target’ students was particularly important for putting the IYT approach into practice. It showed the value of a systematic approach, and related teacher strategies to student behaviour. It gave teachers a heightened awareness of the centrality of their relationship with the student. It also gave tools to identify (and defuse) triggers for negative behaviour, and build positive behaviour in manageable steps.

It was great and very useful, focusing on one aspect of behaviour and building it up slowly. Very specific and reachable goals. Intend to use it in the future. I am a much better teacher and the strategies have helped me grow personally and professionally. Thank you IYT.

When things start to go wrong in class you realise it’s because you are not sticking to the plan or you are neglecting the foundation stages of the pyramid. The behaviour plan is important as you need to think about specific behaviours, specific opposite behaviours and the possible cause — attention/aversion. The behaviour plan has such a positive impact on the classroom atmosphere.

Teachers’ ECE or school contextsMost of the challenges teachers expressed in putting their IYT learning into practice came from their own school or ECE context. Such challenges were expressed by 33 percent of ECE and 29 percent of primary teachers at the end of their IYT programme, and 29 percent of ECE and 20 percent of primary teachers in 2015. Lack of consistency among their colleagues was a particular challenge for ECE teachers, who sometimes were the only teacher in their ECE service to have done IYT. Lack of time was the main challenge for primary teachers.

Our analysis found that gains for teacher practice came when they had active discussion of IYT work with their colleagues, shared that work with others, provided mutual support to make change, and felt supported to make those changes. Simply having done IYT with others in their school or ECE centre was not enough on its own. Teacher aide and RTLB support for IYT approaches was also shown to be useful for making and sustaining change in practice.

In 2015, most teachers reported that they had some support for making or sustaining their IYT-related work within their school or ECE service, though not as much as when they were on the IYT programme.

We also found that teacher practice changed more when IYT was seen to be compatible with the curriculum, as enacted in particular ECE centres and schools. For example, where the New Zealand Curriculum’s (NZC) key competencies

Incredible Years Teacher programmes 9

Evaluation summary

were fully included in a school’s approach to curriculum and its timetabling, IYT was seen and valued as fully compatible with their enacted curriculum.

Our full findings are in these reports:

Wylie, C., & Felgate, R. (2016). 2014 IYT group leaders’ views and experiences: Incredible Years Teacher programmes — Evaluation Report 1. NZCER report for the Ministry of Education.

Wylie, C., & Felgate, R. (2016). IYT in New Zealand: Teacher participants’ reports of learning and change: Incredible Years Teacher programmes — Evaluation Report 2. NZCER report for the Ministry of Education.

Wylie, C., & Felgate, R. (2016). Use of IYT learning in New Zealand: Changes for ‘target’ students and sustainability of learning and practice 8–9 months after the end of the IYT programme: Incredible Years Teacher Programmes — Evaluation Report 3. NZCER report for the Ministry of Education.

10 Incredible Years—Teacher

Evaluation summary

Theory of change for Incredible Years Teacher—short-term outcomes

PB4L base Evidence-based and researched strengths-based programmes or systematic approaches whose ‘delivery’ is consistent with its ‘messages’ (content) Strong emphasis on building teacher and ECE service and school agency (confidence in taking responsibility for shaping the environment promoting positive behaviour rather than seeing behaviour management as largely reactive and limited to consequences), using ongoing data/evidence-based inquiry cyclesStrong emphasis on importance of relationships and sharing of knowledge, and co-constructionNational teams leading and supporting local delivery through formative quality assurance, information and data-sharing and knowledge-building partnership, rather than line management

Alignment Government priorities:Increased achievement of NCEA Level 2 and NZQF level 4Increased participation in ECEGovernment strategies to improve children’s and young people’s wellbeing, and engagement and gains from learning:Ka HikitiaPasifika Education PlanSuccess for AllPriority learner groups (Māori, Pasifika, students with special needs, students from low socio-economic backgrounds)New Zealand Curriculum (particularly the inquiry cycle

Quality of IYT delivery Systematic capability and capacity building and ongoing supervision and monitoring of IYT data trends to build knowledge over time; within and between PB4L national team and partners; within partners (e.g., ‘induction’ of new group leaders—buddying and rehearsal of responses to challenges to aspects of programme from participants)

IYT programme Content (e.g., pyramid; strategies)Way delivered, e.g.: - use of participant self-ratings

and evaluations to customise to participants

- modelling co-construction and agency in relation to problem solving

- inquiry cycle using ‘target child’ and behaviour plan

- coaching between group sessions for individuals also models co-construction, problem solving, use of evidence and review as part of teaching practice

- affirming of participants- culturally responsive- providing safe environment

ECE service or school contextLeaders actively support teachers during the IYT course (e.g., ensuring they have time in the school day for their coaching sessions with IYT group leader) and in their use of IYT afterwards Colleagues in the ECE service/school also use IYTParents take part in IYP Support for use of IYT strategies and/or problem solving from RTLB, Special Education, ERO etcCoherence of curriculum, pedagogy and behaviour

Teacher outcomes:- framework for defining

classroom behaviour management in terms of teacher relationship with and knowledge of each child, and intentional teaching

- modelling and using strategies for children’s self-regulation

- effective strategies to work with disengaged and disruptive students

- assume agency for problem solving, rather than hand ‘problem’ child over to RTLB or school management

- effective relationships with parents and whānau

- sense of greater efficacy - change to teaching practices

Students’ outcomes:- feel respected and

supported in relationship with IYT teachers

- children who were not engaged in learning become engaged in learning

- learn ways to manage themselves and their emotions

- classrooms more focused on learning, fewer behaviour issues or distraction

ECE service or school outcomes:Where a critical mass of IYT teachers, supported by service or school leaders:- strengthened collaborative problem-solving school/service

culture and processes- stable teams - sustained use of IYT in junior school- may also feed changes in teacher practice in senior school

Group leaders’ fidelity to IYT programme

Sufficient time allocated and actually available to group leaders to get ECE manager or school principal backing for teachers’ IYT participation; to plan, deliver, gather, enter and review data, and work with teachers in between group sessions

Course resources available for group leaders and participants when needed