Evaluation and combined methods research

Geoff LindsayARM 31.1.15

• What is combined (mixed) methods research?

• Can it be justified?• Examples of studies• Impact• Conclusions

overview

• Quantitative and qualitative• Designed to optimise strengths, limit

weaknesses• Research question(s) driven

– Major and subordinate

Nature of combined methods

• If only quantitative– Why? Are they satisfied

• If only qualitative– How many? Representative?

• What works but also – reasons – and what context(s)

4

designs• Exploratory

– Focus groups– Identifying the research questions in detail?

• Explanatory– Combination of quant & qual– More complex sequential designs– Convergence– Which is emphasised – quant or qual?

• Eg follow up of quant by quall to identify ‘why?’• Triangulation

– concurrent5

• Complexity of the project– Programme evaluations produce a number

of questions– require a number of methods to address

• Method variation within a paradigm– E.g. Focus groups → individual interviews– Observations → interviews → focus groups

6

teams• Necessary and enhance CM• But need to ensure a real ‘team’ not a

‘collection’ for optimising the research– Parallel, semi-detached approaches can

work if managed well

– Some examples:

7

• Pathfinder 2006-1, national roll-out (PEIP) 2008-11

• Study of parenting programmes for which there is prior evidence of efficacy– Webster-Stratton’s Incredible Years– Triple P– Strengthening Families, Strengthening Communities– Families and Schools Together (FAST)– Strengthening Families Programme 10-14

• Parents with child aged 8-13 presenting or at risk of having behavioural difficulties

8

1. Parenting Early Intervention Programme (PEIP)

Pathfinder• To examine the effectiveness of the roll out of each

programme across LAs in England– The effectiveness of each programme when rolled out– The effectiveness of this delivery model

• To identify implications for future development across the country

• Processes as well as outcomes

Our results led to DfE funding ALL LAs to implement parenting programmes• PEIP

– Repeat for national roll-out – all English LAs 9

Aims of the evaluation

methods• Quantitative

– Parent completed pre- and post-parenting group questionnaires

– Numbers of groups/parents supported• Qualitative

– Over 400 semi-structured interviews with parents, facilitators, strategic and operational leads, head teachers

– 10

Evidence - efficacy• Randomized control trials• Extensive evidence (though varied

across programmes) for efficacy– Outcomes under optimal conditions

• The PEIP provided evidence for effectiveness under real life conditions, on a national scale

Design of PEIP 2008-11• [Pathfinder (Wave 1, 2006-8)] • 47 LAs:

– All 23 Wave 2 LAs, 24 Wave 3 LAs

• Pre-, post- parenting group measures plus 1-year follow up

• Interviews with parents, professionals• Analysis of the programmes

Quantitative• Pre- and post group questionnaires to

parents– Child behaviour

• (SDQ)

– Parent well being• Warwick and Edinburgh Mental Well–being

Scale

– Parenting skills• Parenting Scale

Did PEIP target the ‘right’ parents?• Most (91%) were the biological parents to the target child,

and 85% were female• Wide range of minority ethnic groups (19% vs 8% nationally)• High proportion single parents (44% vs. 24% national

statistics) and two-thirds living in rented property (63% vs. 27%)

• 54% no qualifications or only some GCSEs, but 20% with HE qualifications and 11% with degrees

• High levels of support needs: 49% had seen GP in last 6 months and 21% had seen a social worker

• Low levels of mental well-being: 75% of parents scored below the national median.

Did PEIP target the right children?

• Mean age 8.6 years (but wide range: 37% age 0-7; 54% age 8-13; 9% age 14+)

• 62% boys

• 49% entitled to FSM (16% nationally)

• 12% had statements of SEN (3% nationally)

• 31% reported additional educational support in school

• High % had behavioural, emotional and social difficulties with a negative impact on everyday life (home, friendships, classroom learning, leisure and burden on family) e.g. 61% vs national 13% with conduct problems

SDQ scores at pre-group (% abnormal)

national PEIPEmotional symptoms 11.4 39.2

Conduct problems 12.7 60.7

Hyperactivity 14.7 48.3

Peer problems 11.7 44.7

Total difficulties 9.8 56.5

Prosocial scale 2.3 18.8

Impact score 8.8 60.4

p < .001 in all cases

Did the parent and child outcomes improve after the groups?

• All effects are statistically highly significant

• Effect size is a standardised way of presenting the change in outcomes:

<0.2 small, 0.5 medium; 0.8+ large.

• Parent outcomes show the largest effects because directly targeted

Quantitative data: conclusions• Clear evidence of success of the PEIP

– Parent and child improvements across a range of important dimensions

– Parents very positive about the group experience (those who completed)

– Maintained positive outcomes one year later

The Qualitative data• A total of 429 interviews with LA strategic leads

and/or operational leads; other professionals involved in parenting support,; parenting group facilitators; school representatives & parents

• Changes noted in parents’ own behaviour included:– Setting boundaries and applying them consistently– Giving more praise– Keeping calm and not shouting– Giving instructions to children in clear terms so they

understand what is required– Spending more time in talking to their children– Having more ‘family time’

• Positive changes seen in children’s behaviour included:– Increased confidence– More considerate of other people’s feelings– More calm and more open– Improved attitude towards parents and siblings– More often compliant when asked to do things– Improved school attendance

LA factors & quality outcomes

• Strategic leadership & operational co-ordination combined supported roll-out of the PEIP in LAs.

• Where these were not in place, the PEIP was less efficient in organising groups & reaching parents– Strategic leadership, including the existence

of a parenting strategy, helped establish the PEIP and support it through the roll out

– Strategic leadership meshed the PEIP with existing priorities & infrastructure.

LA factors & quality outcomes

• Operational co-ordination helped PEIP to be delivered across an LA.

• Models differed – some LAs had one central co-ordinator & others divided the co-ordination role on a geographic basis. – However, some central co-ordination enabled LA-wide

oversight of processes & outcomes. – The PEIP LA which returned most pre & post course

CEDAR booklets by far adopted a model of area co-ordinators taking responsibility for their area, but were also linked to a strategic centre with oversight.

Outcomes• Overall PEIP was effective on all our measures

– And improvements were maintained one year later• All 4 programmes were effective

– with some differences between them• Large variation between LAs in numbers of parents supported -

cost effectiveness varied as a result• Average cost per parent was £1658 for those completing, lowest

was £534 in one LA• Important factors identified to guide policy and practice• Very successful government initiative and clear evidence in

support of use of these programmes by LAs

Reference

Lindsay, G., Strand, S. & Davis, H. (2011). A comparison of the effectiveness of three parenting programmes in improving parenting skills, parent mental well being and children’s behaviour when implemented on a large scale in community settings in 18 English local authorities: The Parenting Early Intervention Pathfinder (PEIP), BMC Public Health 2011, 11:962 doi:10.1186/1471-2458-11-962

24