Doctor Natasha Pearce

Making it work: strengthening schools’ implementation of practices to minimisestudent bullying behaviour

Telethon Kids Institute, Australia

Overview

• Why implementation matters • Formula for success • What schools can do to

improve implementation quality of whole-school bullying prevention interventions

What is implementation and why does it

matter?

Need for improved translation efforts

Implementation Science

The research to practice gap

p. 8

Implementation PracticeResearch

Implementation (making it happen): Methods or techniques used to enhance the adoption, implementation and sustainment of an EB program or

practice (Proctor, Powell and McMillen, 2013)

Translation of evidence to practice

What is the problem and

feasible solution/s?

Does it work and can it be replicated in

real world schools?

Can it be effectively

implemented in lots of schools?

Is it working in practice in

enough schools with

impact?

(O’Hara et al, 2013., Woolf, 2008)

T1 T4

Building evidence-based practice

T2 T3

Paradox: To improve evidence-based practice, we need more practice-based evidence!

From EBP to EIP to EBDM…

• EBP is often seen as selecting from ‘lists’ of proven interventions• Broader than this to include:

– Evidenced-based programs/practices– Evidenced-based processes (implementation/service delivery) – Professional (staff) and client (students) needs, experience, values

and beliefs (fit to context) • Evidence-informed decision making uses evidence from all three

sources to make decision about what and how interventions will be used in practice

(Moore, 2016)

Implementation Matters

“…in some analyses, the quality with which the intervention is implemented has been as strongly related to effects as the type of program, so much so, that a well implemented intervention of an inherently less efficacious type can outperform a more efficacious one that is poorly implemented.”

(Lipsey et al, 2010)

p. 11

Implementation matters…

A review of over 500 implementation studies in field of prevention and health promotion programs for children and youth found:• Magnitude of mean effect sizes were at least 2–3 times higher

when programs were implemented well with few problems, particularly when fidelity (as planned) and dose (how much) were used as measures

(Durlak and DuPre, 2008)

Implementation Matters

• Australian study of a whole-school mental health initiative KidsMatter, showed a significant positive relationship between students in high and low implementing schools that was equivalent to a difference in academic performance of up to 6 months of schooling (Dix et al, 2011)

• RCT of Friendly Schools in secondary schools (whole-school, cohort years 8 and 9) found schools differed significantly on bullying victimisation, perpetration and cybervictimisation for students where the SEL curriculum was taught with fidelity (as planned) (Cross et al, in press 2018)

Implementation in Education Findings:

• Only 36 studies where implementation effectiveness/ quality was investigated along with student achievement. Mostly US based, limited strategies (training and coaching)

• Currently limited use of implementation concepts, models and outcomes in education to guide practitioners on how best to deliver educational interventions

(Albers &Pattuwage, 2017)

Barriers and capacity needs: Stakeholder perspectives

• Schools are complex systems in which student wellbeing is a priority but not core business

• National policy mandates schools to have a bullying policy and a framework for action exists but no strategic implementation support

• Each State/Territory varies in its policy focus but largely reactive than preventive e.g. behaviour management

• No tools for identifying and selecting best practice interventions and collectively evaluating impact

• Cross sector collaboration and leadership at a system level critical but coaches/mentoring support to individual schools and teachers needed

(Pearce et al, 2011)

Barriers and capacity needs: School leaders perspectives

• School fear of ownership of bullying behaviours• History of implementing whole-school interventions (burnt out)

and staff and leadership turnover • Schools mostly reactive - preventative approach with practical

tools to save time and resources • Support for leadership teams on choosing evidence-based

practices and how to achieve all staff ‘buy-in’ • Accessible tools to build student, staff and parent competencies

without expensive out-of-school training• Lack of local data to support school decision making• Simple framework to support whole-school change to achieve a

range of wellbeing, behaviour and learning outcomes

(Pearce et al, 2011)

Putting evidence to work: Formula for success

Formula for Success

WHATEffective

Interventions

HOWEffective

Implementation

FITEnabling contexts

Socially significant outcomes

(National Implementation Research Network, 2012)

Formula for Success

WHATEffective

Interventions

HOWEffective

Implementation

FITEnabling contexts

Socially significant outcomes

(National Implementation Research Network, 2012)

What do we know about bullying behaviours in

schools?

Evidence synthesis: What we know about bullying behaviours in schools

• Complex social/relationship issue in schools world wide

• Prevalence: 25% of young people aged 8-14 years in Australia report being frequently bullied (Cross et al., NCBPS, 2009)

• Research into the significant short and long term impacts and the associated risk and protective factors are well established

Evidence synthesis: What we know about bullying behaviours in schools

• Bullying behaviours peak in Year 5 (social development) and year 7/8 (transition) into secondary school

• Covert bullying, in particular cyber bullying, although less prevalent, has the potential to cause more harm.

What do we know about bullying prevention

interventions in schools?

Delivery balance for bullying prevention

Prevention

Intervention

Treatment

Whole school environment promoting competence, health and wellbeing ≈ 80%

Students with high support needs ≈ 15%

Students needing additional intervention ≈ 5%

(O’Connell et al, 2009)

Bullying prevention interventions

• Most of the bullying prevention programs that have been rigorously evaluated have used a universal approach to address the complexities of these social behaviours and the need to address multiple outcomes (behaviour, wellbeing, mental health etc) (Bradshaw, 2015)

• These programs may aim to improve school climate, shift norms about bullying, mobilise bystander behaviours and change the physical school environment

• A whole-school approach works - multi-dimensional prevention and targeted strategies at all levels (school, classroom, home and individual) (Farrington & Ttofi, 2009)

SEL Interventions• A meta-analysis of 213 school-based, universal social and

emotional learning (SEL) programs involving 270,034 kindergarten through high school students. Compared to controls, participants demonstrated significantly improved social and emotional skills, attitudes, behavior, and academic performance that reflected an 11-percentile-point gain in achievement. (Durlak et al, 2011)

Bullying Prevention Interventions

• Meta-analysis of 44 studies: can achieve a 20-23% reduction in rates of bullying perpetration and a 17-20% reduction in rates of victimisation (Farrington & Ttofi, 2009)

• But…some mixed results - Reviews of 26 (Vreeman & Carroll, 2007) and 16 (Baldry & Farrington, 2007) school-based bullying interventions - 50% had positive outcomes – other half mixed, small and negative effects

• BUT reviews stated low effect sizes most likely due to implementation and capacity issues (Rigby et al, 2008)

• A lot is known about barriers and enablers to implementation butless evidence of the links between implementation strategies and impact on student outcomes (Farrington & Tofi, 2009, Pearce et al,2011)

Multiple outcomes

Formula for Success

WHATEffective

Interventions

HOWEffective

Implementation

FITEnabling contexts

Socially significant outcomes

(National Implementation Research Network, 2012)

Implementation is a process not an event

Stages of implementation Successful implementation takes time

2– 4 Years

Exploration Prepare Deliver Sustain Explore

(EEF, 2018; NIRN, 2012)

Friendly Schools Implementation Process

Stage Stage 1: Explore Aim: To define the problem and fit to school community needs• Explore school needs using local and system data • Specify a focus for improvement - is it amenable to change?• Select interventions proven to work – can some less effective

practices go? Be strategic and implement well – less is more!• Is it well defined - clear core components, operational

definitions (what does it look like in practice) and practical performance assessment

• Is it useable - practical for schools in delivery time and capacity? • Assess fit and feasibility to the school context • Make an adoption decision

Macro policy and frameworks

Staties

Aim: To build the infrastructure and capacity needed to create a enabling environment to support change• Develop a logical implementation plan and execute it in stages: explore,

prepare, deliver and sustain • Build leadership capacity through implementation teams • Specify core components – what can be adapted, what cannot? • Assess staff feelings of readiness before implementing an intervention • Identify barriers and enablers to implementation and match strategies

that work at the staff, team and school level • Identify clear implementation outcomes and practical measures • Train staff in the active ingredients of the intervention with initial

training, follow-up coaching and ongoing peer collaboration support

Stage 2: Prepare

Stage based activities

Aim: To support staff, monitor progress, solve problems and adapt strategies as it is used for the first time• Adopt a flexible leadership approach • Follow up support with staff – with skilled coaches and mentoring or peer-to-peer

collaborations • Use implementation data to assess progress and inform when adaptions may be

necessary like fidelity (implemented as planned), acceptability, feasibility, reach and cost

• Only make adaptions once the core components are securely understood and implemented

Learn from mistakes, continue buy-in and manage expectations

Stage 3: Deliver

es

Aim: To plan for sustaining and scaling from the outset and continually learn and nurture its use • Plan to scale the intervention efforts from the outset and

treat it as a new process with potential new barriers • Maintain and improve skills and activities • Ensure implementation data remains fit for purpose• Be ready to evaluate expected outcomes! • Continuously reward good implementation practice

2–4 years for sustainability and 5-7 years institutionalisation

Stage 4: Sustain

Implementation Road Map

Changing on purpose…

• New programs/practices are not likely to survive in existing systems…existing systems need to change to support the effectiveness of the intervention – adaption to context is necessary for quality implementation!

• Improvement cycles create a learning culture• In a complex system the solution will change the problem

– need to see what emerges over time• Allow enough time for effective implementation (2–4yrs),

especially in preparation stage

Top 5 Tips What schools can do to improve

implementation quality of bullying prevention interventions

Top 5 key learnings from FS schools - 1

• Do less more effectively! Use one strategic framework for student wellbeing/learning that maps a collaborative, coordinated, whole-school approach using developmentally appropriate evidence-informed strategies. Remove ones that don’t fit.

”…you can take on programs and do them as add-ons but they don’t change the culture of the school…you really need to make them the pastoral care fabric in the school…” (School Principal).

“To try and improve transparency and collaborative efforts of everyone…to make a more coordinated approach, almost a

course of study that can be…followed across 6 years” (School Leader).

41

Top Top 5 Key learnings from FS schools - 2

• Adopt a systematic implementation process that builds capacity through improving staff competencies, organisational support and leadership.

”I think there is probably greater self-confidence in our capacity to make change…I think there is an increased perception that we can actually challenge behaviours and

stereotypes” (School leader).

“ I think the learning is getting it well embedded at the top with an empowered group responsible for implementation. Get all that space really clear from the beginning”

(School Leader).

42

Top Top 5 Key learnings from FS schools - 3

• Implement evidence-informed strategies that are driven by local school data and context over a realistic timeframe (2 to 4 years) to achieve positive social and emotional wellbeing that supports academic learning.

“We don’t have the answers for other schools because it’s all about context…its not easy, it takes a long time. Its got to be central and integral. If you are dealing with it

only once the issues have happened…then you have lost. Our student support team job descriptions are about positive stuff…relationships…helping students.. not dealing with

problems” (School Leader).

43

Top 5 Key learnings from FS schools - 4

• Build pastoral care/student support as an integral part of the school administration (eg. year or house leaders) with dedicated time for collaborative planning and delivery.

“…we’re blurring the lines between academic staff and pastoral care…” (School Leader)

“The student support role from K–12 has been a real eye opener for the leadership team…hitting those behaviours in primary school makes a massive difference to

students coming into high school…” (School Leader)

44

Top 5 Key learnings for FS schools - 5

• Adopt student support or pastoral care systems and activities that facilitate cross-year group relationships with a designated location in the school where all students can access a range of health, career and psychological supports.

“The student services office is not a withdrawal centre…its a place for kids to go to for pastoral care and relationship breakdowns are issues to deal with, they’re not things to

punish” (School Leader).

45

Takeaways for high quality use of bullying prevention interventions es

Changes in significant outcomes for students requires:• Effective, well-defined interventions• A change process that embeds and sustains the

intervention in a real-world school (stage matched activities and drivers to build infrastructure and capacity)

• Improvement processes that enable the context tosupport the effective intervention

• Implementation teams to make it happen! • Supportive system: schools’ implementation of EIP needs to be supported

with pre-service teacher training in those EIP and professional learning and mentoring for existing teachers already working in schools

Resources

Resources

https://educationendowmentfoundation.org.uk/tools/guidance-reports/a-schools-guide-to-implementation

Resources

• http://evidenceforlearning.org.au/the-toolkit/

Thank you to all Friendly Schools for their participation in research