BuildiNg childreN’s coNfideNce aNd improviNg pareNts’ … · 2020-01-09 · Children valued the opportunity to meet with peers facing similar difficulties to themselves, and learning

Impact and Evidence series

November 2017

BuildiNg childreN’s coNfideNce aNd improviNg pareNts’ protective skills

fiNal evaluatioN of the Nspcc family smiles serviceRachel Margolis and Prakash Fernandes NSPCC Evidence team

Impact and Evidence series

This report is part of the NSPCC’s Impact and Evidence series, which presents the findings of the Society’s research into its services and interventions. Many of the reports are produced by the NSPCC’s Evaluation department, but some are written by other organisations commissioned by the Society to carry out research on its behalf. The aim of the series is to contribute to the evidence base of what works in preventing cruelty to children and in reducing the harm it causes when abuse does happen.

©2017 NSPCC. Photography by Tom Hull. The people pictured are models.

3Impact and Evidence series

Contents

Impact and Evidence series 2

ACKNOWLEDGEMENTS 5

ExECuTIvE SuMMAry 6

MAIN rEPOrT 11

Chapter 1: Introduction 111.1 Background 11

1.2 Theory of change 14

1.3 Evaluation aims and methodology 18

Chapter 2: Outcomes for children and young people 292.1 Change in emotional and behavioural problems 29

2.2 Change in children and young people’s self esteem 35

2.3 Change in children’s ability to process their thoughts and feelings 36

2.4 Factors affecting outcomes for children and young people 38

Chapter 3: Outcomes for parents 503.1 Parents’ perceptions about their parenting and impact of

mental health problems 50

3.2 Change in behaviours that contribute to risk 52

3.3 Examples of changes for parents 53

3.4 Factors affecting outcomes for parents 57

Chapter 4: Exploring sustained change 664.1 Sustained changes for children and young people 66

4.2 Sustained changes for parents 68

Chapter 5: Conclusion and discussion 715.1 Suggestions for improvement 72

5.2 Discussion 73

Bibliography 78

Building children’s confidence and improving parents’ protective skills4

APPENDICES 81Appendix 1: Programme details 81

Appendix 2: Interview schedules 85

Appendix 3: Ethics Overview 91

Appendix 4: Standardised measures 93

Appendix 5: Characteristics of children and parents at each point of the referral journey and the evaluation 95

Appendix 6: Statistical analysis and qualitative data management 97


ACKNOWLEDGEMENTSThe authors would like to thank:

• The children and parents who consented to take part in the evaluation, for giving their time to complete questionnaires throughout the programme.

• Children and parents who consented to take part in the interviews and share their perspectives and insights about the programme.

• Team managers, practitioners and administrative staff at the Family SMILES service centres in Belfast, Bristol, Coventry, Gillingham, Grimsby, Ipswich, Jersey, Manchester, Middlesbrough, Prestatyn, Southampton and Tidworth, who supported the implementation of the evaluation by explaining its importance to children and parents and ensuring that data was entered onto the system for analysis.

• Di Jerwood, Neil Peake, Sue Walls, Tom Rahilly, Peter Richards, Fiona Richards and Julie Taylor for their advice, suggestions, encouragement and promoting the evaluation within the service.

• Matt Barnard, Richard Cotmore and Mike Williams for reviewing the evaluation plans and reports.

• Helen Fisher, for her help and support with the statistical analysis of the evaluation data.


executive summAryBackgroundFamily SMILES (Simplifying Mental Illness plus Life Enhancement Skills) is an intervention for families with children who have been identified as being at risk due to (one of) their parent’s mental health problems. It aims to reduce the negative impact of parental mental health problems on children and ensure that they are kept safe. It consists of eight weekly group sessions for children, six individual sessions for the parent and a joint session for the parent and child together. An assessment of up to four sessions takes place prior to this to determine eligibility for the programme.

This final report is based on the evaluation data collected since the project began in September 2011 until December 2015. During this time, the service was run at 15 NSPCC sites (12 of which were part of the evaluation), and in total 191 parents and 333 children completed the programme. The evaluation is based on the experiences of 59 parents and 230 children.

Aims and methodology The evaluation of Family SMILES sought to evidence whether the following key outcomes were achieved for children and young people: increased self-esteem; reduced emotional and behavioural difficulties, and improved ability to process thoughts and feelings. There were two key outcomes for parents: greater insight into the impact of their mental health problems on their child and enhanced protective parenting behaviour. It was envisaged that these changes would strengthen the parent-child relationship by improving communication within the family and so contribute towards keeping children safer, as conceptualised in the theory of change that was specifically developed for the programme (see Chapter 1.2).

The key elements of the evaluation design included:

• An impact evaluation, using pre-, post- and follow-up measures to gather quantitative data from the perspectives of children, parents and practitioners. Quantitative findings were described in terms of their statistical significance; whether there was a clear trend in changes in average scores of the sample, and clinical significance – whether scores crossed thresholds defined by developers of the standardised measures relating to clinical need.


• A comparison group formed from a naturally occurring waiting list. This group was made up of families who had gone through the assessment for the programme but were waiting to be allocated to an appropriate group.

• Qualitative interviews with samples of children, parents, practitioners and referrers. The interviews aimed to explore perceptions and experiences of the Family SMILES programme; outcomes, and factors that helped or hindered the achievement of those outcomes.

Key findings

Positive outcomes for children and parents by the end of Family SMILES

There was a small but statistically significant improvement in children’s emotional wellbeing and behaviour, according to the key outcome measure used, the Strengths and Difficulties Questionnaire (SDQ). The shift in the proportion of children who had the highest (clinical) levels of need at the beginning of the programme to a lower (non-clinical) level at the end was statistically significant, indicating that this is unlikely to have happened by chance. The changes were greater for children who had taken part in the whole programme than for those in the comparison group. More specifically, there were significant improvements in children’s conduct and hyperactivity at the end of the programme. There was also a small but statistically significant improvement in children’s self-esteem and ability to process their thoughts and feelings.

Qualitative findings highlighted what some of these changes looked like for children: feeling more confident and able to engage in a range of new and different activities; being more able to express emotions and concerns to trusted adults, and feeling less anxious and more reassured.

There were statistically significant improvements in parents’ protective parenting that were not found in the comparison group, though their high levels of unhappiness and distress (according to parents’ scores on the Child Abuse Potential Inventory) reflected ongoing struggles with mental health problems. Nevertheless, interview data highlighted improvement in parents’ abilities to communicate with their children and changes in their parenting.


Changes to children’s emotional difficulties: a matter of perspective?

SDQs were completed by either the child or their parent. Parents reported a significant improvement in children’s emotional wellbeing, which crossed the threshold from the highest to a lower level of concern. However, children did not share this perspective and did not report a significant improvement in this area. This suggests the possibility that parents and children perceive changes differently or that parents were responding to observable changes that were not matched by children’s inner experience.

Mixed findings around sustained change

Data collected to explore children’s emotional and behavioural difficulties and self-esteem six months after completing Family SMILES suggested continuing improvements. However, these changes were not statistically significant and hence it is not possible to state conclusively that they were sustained beyond the end of the programme. The sample size of parent data at the time of six-month follow up was too small for undertaking statistical analysis around sustained changes.

Facilitators and barriers in achieving change for families

Children valued the opportunity to meet with peers facing similar difficulties to themselves, and learning about mental health. The group work also helped them to believe that they were not to blame for their parents’ mental health problems. Children benefitted from the child-centred model of Family SMILES, which brought children’s needs to the fore in an area of adult mental health provision where they can often be overlooked. Parents placed importance on having the opportunity to reflect on the impact of challenging events they had experienced in life. They also benefitted from the strengths-based approach taken by practitioners, which helped them to recognise their positive qualities as a parent. Gaining an insight into their children’s perspective on their mental health problems; developing a support plan as a family, and the flexible and reassuring approach of practitioners were also identified as helpful aspects of the programme.

Barriers experienced by some children included their limited understanding of mental health at the outset of the programme; the challenging behaviour of other children in the group, and the struggle to cope with the group work ending. Other barriers were associated with children using information about mental health inappropriately – for example, to cause upset at home. Parental concerns around children learning about mental health also acted as a barrier. For parents, barriers to achieving outcomes included their inability to


acknowledge their mental health problems; ongoing problems in the parent-child relationship, and complex family circumstances – such as seeking asylum or coping with poor living conditions.

Bridging the gap between adult mental health and children’s services

The evaluation highlights the value of an approach to service delivery that works with the whole family, bringing together adults and children who may previously have been engaged with multiple agencies. This suggests that Family SMILES can bridge a gap between adult mental health and children’s services by creating a safe space for parents and children to explore the impact of mental health on the family. The capacity of the programme to bridge that gap was however undermined by not being sufficiently integrated with the adult and children’s services. A failure to adhere to the inclusion criteria – which required all participating parents with a mental health problem to be receiving mental health support – may have exacerbated this.

Improving communication within the family

Families described how the programme had given them the language to talk about mental health with each other. This included a greater knowledge about terminology (medication, treatments and conditions) and their meanings, which enabled communication about these topics to be more appropriate. In addition to language, children suggested that having a better understanding of their parent’s mental health problem also helped them to respond confidently and appropriately when their parents were experiencing difficulties.

Programme design is key for enabling families to achieve outcomes

Participants reflected that the structure of the intervention – giving children their own space to discuss issues in a group work setting; working with parents individually for them to process their thinking, and the possibility of involving the entire family – were all important elements of the programme. The group work element of the programme was particularly valued by children. Where children struggled to come to terms with the group ending it was suggested that some form of ongoing support, in the form of group work, might be helpful.


Delivery of the programme

Parents, children and practitioners commented positively on the flexibility of the programme, in terms of children having the opportunity to do one-to-one sessions alongside the group work; parents having the space to talk about issues currently affecting them, and the additional support provided by NSPCC practitioners in supporting families – especially those where children were on a child protection plan. Any transfer of the Family SMILES model needs to take into account this degree of flexibility and the engagement skills and approach of NSPCC workers that were highly valued by parents and children.

Implications

The theory of change for the Family SMILES programme needs to be reviewed to reflect a clearer understanding of broader structural and societal factors that can impinge on parental mental health and a family’s lived experience. It should also articulate the assumptions relating to what needs to be in place to generate change, for example the formal support from mental health services – is it acceptable to include parents who do not have such support?

If this programme is run again, it should be more closely integrated with other systems to maximise its potential for bridging the adult and children’s services sectors.


mAin report

Chapter 1: IntroductionThe Family SMILES (Simplifying Mental Illness plus Life Enhancement Skills) programme is an intervention for families with children who have been identified as being at risk due to (one of) their parent’s mental health problems. It aims to reduce the negative impact of parental mental health problems on children and ensure that they are kept safe. This final report is based on the evaluation data collected since the project began in September 2011 until December 2015. This introductory chapter summarises Family SMILES and describes the evaluation design.

1.1 BackgroundRecent reports suggest that there are many families in the UK where one or both parents experience mental ill health. For example, researchers have found variously that:

• more than one third of all adults in the UK who experience mental ill health are also parents (Aldridge, 2011)

• there are more than 4.2 million parents who experience mental health problems (Crossroads Caring for Carers and The Princess Royal Trust for Carers, 2008)

• 9–10 per cent of women and 5–6 per cent of men are parents with mental ill health (Henshaw, 2011)

• of those adults accessing mental health services, it is suggested that around half have children under the age of 18 years (Bee et al, 2014).

Families where the parent has a mental health problem can have complex needs, and without effective support, these can have negative effects on the children. These may include low self-esteem, stress, missing school, behavioural disturbances, inappropriate sexual behaviour, and the development of mental health problems such as eating disorders or depression (Aldridge, 2006; Stallard et al, 2004; Tunnard, 2004). In the worst-case scenario, children can be seriously harmed or even die – parental mental ill health is a common factor in child protection investigations (Prokop et al, 2010): over half of the 33 serious case reviews in England from 2009 to 2010 involved parental mental health problems. Referring specifically to children aged 5–10, Brandon et al found that: “parental mental health problems featured in a majority of cases, and suicidal or self-harming behaviour was particularly prominent.” (2011:5).


Parental mental ill health does not necessarily lead to problematic parent-child relationships or poor outcomes for children, however. Aldridge (2011:2) considered factors associated with poor outcomes and concluded that for children:

“…both caring and adverse consequences of living with a

parent with mental illness are likely to occur in families where the

impacts of social exclusion are profound or combined (impact of

low income, poverty, poor housing, etc).”

One implication of this is that professionals need to take a wider perspective to include broader structural and societal factors that may underpin the parental mental ill health and the family’s lived experience:

“…they should also acknowledge the impact of social exclusion

factors such as poor housing, poverty, etc. this should include

understanding more about the family’s location, status and

experiences in their local neighbourhood or community” (ibid:3).

In addition, many children have positive outcomes or at least find ways of coping well:

“in family situations where children are accorded a degree of

control and have secure family relationships with others – either

aside from or including the affected parent – then children can

develop resilience and effective coping strategies” (ibid: 2).

A range of professional practice challenges arises not just from the complexity of families’ needs but also from the organisation of services. Specialisation between children’s and adult services has generated benefits, but has also served to create a gap between services in many areas and limited the ‘breadth of view’ of some professionals and organisations (Roscoe 2012). Darlington et al (2005) identified mutual distrust between mental health and child protection workers, fuelled by a range of factors that included lack of time and resources, unrealistic expectations, confidentiality issues, and professional domains and boundaries, as barriers to interagency working.

An NSPCC report on what was learnt from serious case reviews that involved parents with mental ill health highlighted poor practice around assessment. This included the need to consider more fully

“…the impact of the stresses of parenting on the parent’s mental

health [and] the impact of the parent’s mental health problems on

the safety and wellbeing of the child.” (nspcc, 2015 online)


In addition, practice was not sufficiently child-centred:

“A number of case reviews highlight that the child wasn’t given

the opportunity to be heard. professionals must talk to the child

and respond to what they say.” (ibid)

There have been many promising initiatives to address these practice challenges, especially relating to the gap between adult and children’s services. One was the Think family programme developed by SCIE. An evaluation of that programme across England and Northern Ireland found many strategic and practice initiatives involving restructuring, new assessment tools, workforce development, and greater involvement of service users. But in spite of these and the learning about factors that facilitated change, many problems remained: ongoing challenges in sharing information; a lack of shared-outcome indicators, and a lack of integration with some existing frameworks (for example, the Care programme approach) (Roscoe et al, 2012).

Given all of these challenges, it is perhaps not surprising that there is a lack of evidence of effective community-based interventions aimed at improving or maintaining quality of life for children of parents with serious mental health problems. One systematic review highlighted the lack of evidence on which to base UK service development or delivery (Bee et al, 2014). This evaluation study of Family SMILES seeks to contribute to closing that gap.

Family SMILES is a group work intervention for children whose parent has a serious mental health problem, where the child is at risk of potential harm. It aims to reduce the risk of harm to children living with parents who have mental health problems and improve their self-esteem, resilience and life skills. It aims to help parents understand the impact of their mental health problems on their child and improve protective parenting skills.

Family SMILES is based on the Australian SMILES programme, developed in 1997 by Erica Pitman (Pitman, 2010). The Australian SMILES originally involved group work with young carers who had a parent or sibling with a mental health problem. It has been used in other countries, including Canada, and positively evaluated in a number of reports (Pitman and Matthey, 2002; Baldwin and Glogovic, 2010). Family SMILES is the NSPCC’s adaptation of the original SMILES intervention, taking a more family-based approach (working with both the child and parent) and focusing on child protection issues. It is targeted at children who are vulnerable and have been identified by professionals as being at risk due to their parent’s mental health problems. Appendix 1 provides an overview of the sessions with children and parents. The children may be living at home or placed elsewhere, but will have ongoing contact with the


ill parent. After receiving a referral, the NSPCC carries out a four-week assessment of the family to check their eligibility for the service. Appendix 1 outlines the inclusion criteria for the service.

The Family SMILES model involves group work with children and individual sessions with parents. There are eight group sessions, held over an eight-week period, with a maximum of eight children (8–13 years) per group, combining play, creative activities, discussions and age-appropriate education about mental health and life skills (eg, what to do in an emergency) to help them cope more effectively. Family SMILES practitioners also run up to six individual sessions with parents to help them understand the impact of their mental health problems on their child, and to strengthen the parent-child relationship. If this is not possible, (eg, if the parent is too ill) they will work with the child’s ‘well’ parent or carer. The parent or carer and their child are then brought together at the end of the programme to draw up a support plan to ensure that, in the event of parental relapse, the child will be adequately protected and cared for, and that their wishes will be considered.

The NSPCC runs the Family SMILES programme in Belfast, Bristol, Coventry, Gillingham, Grimsby, Ipswich, Jersey, Manchester, Middlesbrough, Prestatyn, Southampton and Tidworth.2

1.2 Theory of changeThe theory of change conceptualises the programme in terms of inputs, activities, the ways in which the programme helps and the outcomes it aims to achieve.

Key outcomes for children and parents

As outlined in Figure 1, the key primary outcomes of Family SMILES are for children and young people to have more self-esteem, reduced emotional and behavioural problems and an improved ability to process their thoughts and feelings. The key outcomes for parents are: a greater insight into the impact of their mental health problems on their child, and enhanced protective parenting behaviour. These primary outcomes should lead to children’s improved wellbeing and safety.

2 The Family SMILES programme was also implemented in Birmingham, East London and Catterick in late 2015.


Figure 1: Theory of Change

Inputs Activities How the programme helps Outcomes Goal

• Skilled practitioners

• Programme and workbooks

• Local partnerships eg Children’s Services and Adult Mental Health Services

• Improved protective parenting

• Greater insight into impact of mental health problems on children

Jointsession for parent and child

• Sharing experiences and learning from programme

• Practicing open communication

• Jointly participating in support planning

• Learning about parenting and boundary setting

• Learning about the child’s experience of parental mental health problems

• Gaining an insight into own mental health problems and impact on child.

Parents:Sixindividual sessions with the practitioner

• Listening and sharing experiences with other children in a safe, confidential environment

• Reducing isolation• Learning about mental

health• Learning to stay safe

and identify trusted adults

Children:Eightweekly group sessions

• Increased self-esteem

• Reduced emotional and behavioural difficulties

• Improved ability to process thoughts and feelings To improve

the wellbeing of children and young people and ensure that they are kept safe

This section draws on evidence that highlights the pertinence of these outcomes for children and parents from households where there are parental mental health problems.

Children’s self-esteem

Tunnard (2004) has highlighted that children of parents with mental health problems are significantly less happy about themselves than their peers: they are more likely to have lower self-esteem and to feel depressed. Furthermore, as parents may struggle to pay more attention to their child rather than themselves, children can be left feeling isolated and rejected, further affecting their sense of self-esteem (Roy, 1990).

Children’s emotional and behavioural difficulties

Parents with mental health problems have reported concerns about their children’s behaviour, including aggression, inappropriate sexual behaviour and problems with peers (Stallard et al, 2004). Children may experience high levels of anxiety as they worry about their parent’s health and try to cope with the unpredictability of their parent’s consequent behaviour (Tunnard, 2004; Dearden and Becker, 2000). Feelings of embarrassment about their parent’s health may


contribute to a sense of unease around inviting friends to the family home, or causing them to distance themselves from their friends, further affecting the child emotionally (Tunnard, 2004).

Children’s ability to process thoughts and feelings

A lack of understanding of their parents’ behaviour due to an absence of appropriate information about mental health can leave children feeling confused and unable to process their thoughts and feelings about what they are experiencing (Tunnard, 2004).

Protective parenting behaviours

Parents with serious mental health problems have been reported to provide a lower quality of stimulation for their children; to demonstrate less confidence in their parenting, and to display less emotional availability to their children compared to parents who are not experiencing such difficulties (Wan et al, 2007). Parents may also use overly negative or critical language in their parenting and have greater levels of conflict with teenage children (Tunnard, 2004).

Parental insight into impact of their mental health on children

Adults with mental health problems can distance themselves from their children intentionally, in an effort to protect them from their difficulties (Tunnard, 2004). They may struggle to recognise the importance of talking to their children about how they are feeling, communicating with them about their specific condition, and listening to their child’s concerns about what they are experiencing.

How Family SMILES seeks to help children and parents to achieve better outcomes

The ways in which the Family SMILES programme seeks to enable children and parents to improve against these outcomes are outlined below.

Group work for children

Listening and sharing experiencesBeing in a peer group provides children with the opportunity to realise that they are not alone with their problems, to feel safe, and to distance themselves from their difficulties for a while (Dearden and Becker, 1995). Through talking to and listening to their peers, they are able to have fun, build new friendships and develop their sense of self-esteem.


Reducing isolationMayberry et al (2006) have highlighted the importance of children having social and emotional connections and support, both within the family and beyond. By being part of a group, children are provided with the opportunity to engage with others and develop such connections, so reducing any feelings of isolation.

Learning about mental health problemsLearning about mental health helps to address children’s anxieties around how their parent became ill, understanding their symptoms and the treatment that they receive (Stallard et al, 2004). It enables children to realise that they are not to blame for their parent’s mental health problems, and that they cannot prevent their parent from being unwell (Stallard et al, 2004). This provides children with reassurance and helps to address the emotional impact of living with a parent with mental health problems.

Learning to stay safe and identify trusted adultsBy providing children with age-appropriate information on how to keep safe at home, children are more able to understand the circumstances under which, for example, they might need to call a doctor for help or to identify unusual changes in their parent’s behaviour (Tunnard, 2004). The group work helps children to identify trusted adults who they can talk to and gain support from for negotiating complex situations, such as talking to others about their parent’s mental health, which enables them not only to process their thoughts and feelings, but also to express those thoughts and feelings appropriately (Tunnard, 2004).

Individual work with parents

Learning about parenting and boundary setting Adults with mental health problems do not necessarily lack parenting skills, but may struggle with a reduced sense of self-efficacy and confidence in their parenting abilities; this can affect their parenting, particularly in times of stress (Gelkopf and Jubotaro, 2013). With the right support and resources, many parents with serious mental health problems are able to provide appropriate care to their children (Mayberry et al, 2006). The parenting work therefore plays a key role in addressing issues around confidence building and effective approaches in parenting.

Learning about the child’s experience of parental mental health problemsParents are not always able to recognise or acknowledge the ways in which their mental health problems are adversely affecting their children (Stallard et al, 2004). Gaining an insight into their children’s perspective provides parents with an improved understanding around how their behaviour affects the emotions and behaviour of their


children, and an opportunity to explore ways in which this situation can be better managed.

Gaining an insight into own mental health problems Working with professionals who recognise the link between past life experiences, such as domestic violence or loss, with current mental health problems is often valued by adults (Tunnard, 2004). Being able to explore such issues with a practitioner helps parents to gain some insight into their mental health problems, to feel listened to and supported so that outcomes for their families may be improved.

Joint work

While parents and children tend to retreat into their own spaces within the home to try and cope with the impact of a parent’s mental health problems, this is often to the detriment of all individuals involved (Tunnard, 2004). Developing a shared understanding of how best to respond to one another in times of crisis is an effective way of supporting families to cope. In particular, children appreciate being included in assessments and discussions relating to their parent’s mental health (Tunnard, 2004). At the end of Family SMILES, parents and children are involved in formulating a support plan, so that they are better able to communicate and take into account everyone’s needs and wishes.

1.3 Evaluation aims and methodologyThe key questions that the evaluation sought to answer were largely outcome-focused. So in terms of outcomes for children and young people:

1) Has their self-esteem improved by the end of Family SMILES?

2) Have their emotional and behavioural problems improved?

3) Are they better able to process their thoughts and feelings?

And for the parents:

4) Have protective parenting behaviours improved by the end of Family SMILES?

5) Do they have a greater insight into the impact of their mental health problems on their children?


For both parents and children:

6) Are outcomes sustained beyond the end of the programme?

7) What are the key facilitators and barriers to achieving these outcomes – as set out in the theory of change?

Family SMILES was evaluated using a mixed-method, quasi-experimental design that included:

• a pre- and post-test element to examine the extent to which the programme’s intended outcomes were achieved

• a comparison group to help understand the impact of the intervention

• a six-month follow up to gain an insight into sustained change for children and parents

• qualitative interviews to identify the key facilitators and barriers to change.

The comparison group for the evaluation drew on a naturally occurring waiting list of children and parents who had been referred to the programme but who had been waiting a minimum of eight weeks to be allocated to an appropriate group. It was important from a service perspective that families were not asked to wait to start the programme if there was a group ready for the child to join, so the comparison group only consisted of those who had no choice but to wait. The comparison group was made up of 31 children and 22 adults. While the gender balance of the comparison group was similar to that of the pre-/post-intervention group, it was made up of a slightly younger cohort of children, with a smaller proportion of children aged 12–13 than the intervention sample.

It should be noted that the comparison group was not introduced at the start of the study but at a later point. As a result, it is very small sample and caution should be applied in using this data to draw firm conclusions about the impact of Family SMILES. It is also important to note that comparison group families were assessed while on the waiting list. The comparison is therefore between families that had received the assessment plus intervention and those who had received only the assessment – not between families who received support and those who did not.


Impact evaluation

Evaluation measures

The evaluation measures used to quantify change for each outcome identified in the theory of change are summarised below; a more detailed overview of these measures can be found in Appendix 4.

The Strengths and Difficulty Questionnaire (SDQ) (Goodman, 1997)The SDQ measures the level of emotional and behavioural problems of children and young people. The higher the total score, the higher the level of difficulty. Four scoring bands were defined for the SDQ: very high; high; slightly raised, and close to average. In order to determine clinically significant change, the very high and high scoring bands were used to describe clinical difficulty in this report, and the remaining two bands to indicate non-clinical difficulty. Young people aged 11 years or over as well as younger children who practitioners felt were able to complete the SDQ themselves were offered the self-complete version of the questionnaire. The SDQ was completed by the parent/carer for children younger than 11 or for those unable to complete it themselves.

Health of the Nation Outcome Scales Child and Adolescent Mental Health (HoNOSCA) (Royal College of Psychiatrists, 1996The HoNOSCA captures practitioners’ perspectives on children’s behavioural and emotional difficulties. Practitioners provided a score between 0 and 4 for each of 13 criteria set out in the HoNOSCA covering four broad categories: behavioural problems; impairment; symptomatic problems, and social problems.

Rosenberg Self-Esteem Scale (Adapted)The Rosenberg self-esteem questionnaire (Morris, 1981) was adapted by the NSPCC for use with children. Total scores range from 0 to 30, with a higher score indicating a higher level of self-esteem.

Child Abuse Potential Inventory (CAPI) (Milner, 1986)The CAPI was used for Family SMILES as an evaluation tool only and not for predicting abuse. It is a reliable tool for measuring change in protective parenting, containing six validated subscales relating to the parent’s attitudes and interpersonal problems. The CAPI also has inbuilt validity scales to filter out ‘fake’ good responses, inconsistencies or random responses. It consists of 160 questions, with a cut-off score of 215 defining the threshold for poor protective parenting behaviours.

Evaluation WheelsEvaluation wheels were completed by parents and children, who rated themselves between 1 and 5 (1 being low; 5 being high) against 5 criteria. These measures were not standardised.


Service and evaluation attrition

A total of 677 parents and 679 children were referred to the Family SMILES programme, although only 191 parents and 333 children completed the programme in full. A further 84 parents completed the Family SMILES work appropriate to them, but not necessarily the whole programme – for example, some completed fewer sessions, covering only those issues practitioners identified as relevant to that family. Similarly, a further 64 children completed the Family SMILES work that had been agreed with their practitioner, but not the whole programme. Some of these children did not complete the joint work with a parent or carer, or did some of the sessions individually with a practitioner rather than in the peer group. An overview of case progression from referral to programme completion via assessment and starting the programme is represented below in Figure 2.

Figure 2: Analysis of referrals (Parents N=677, Children N=679)

70 (67%) of children did not want to proceed their referral, 29 (28%) were

not eligible for the service

51 children (80%) disengaged from the service.

The service was no longer appropriate for 7 children (11%) e.g. due to changes in care arrangements

54 children (53%) disengaged from the assessment process, 32 (31%) were assessed as ineligible for the service.

e.g. due to lack of knowledge of parental mental health problems

76 parents (55%) disengaged from the assessment process, 50 (36%) were

assessed as ineligible for the service e.g. due to complex mental health needs

67 parents (80%) disengaged from the service.

The service became inappropriate for 11 parents (13%) e.g. due to deterioration in mental health

84 parents (23%)

64 children (14%) dropped out of the

service early

281 parents completed the one-to-one work

408 children completed the group work

365 parents began the one-to-one work

472 children began the group work

502 parents assessed

574 children assessed

191 parents completed the programme in full (52% of those who started, 28% of those who were referred). 90 parents completed

the work agreed with the practitioner, but not the full programme.

333 children completed the programme in full (71% of those who started, 49% of those who were referred). 75 children completed

the work agreed with the practitioner, but not the full programme.

175 parents (26%)

105 children (15%) not invited for assessment

128 parents (73%) did not want to proceed their referral, 39 (22%) were

not eligible for the service

137 parents (27%)

102 children (18%) had their cases closed

during assessment

677 Parents referred

679 Children referred


Figure 2 indicates that 52 per cent of parents who started Family SMILES went on to complete the programme in full, representing 28 per cent of parents who were referred to the service. This indicates relatively high levels of attrition for parents. Levels of attrition were much lower for children, with 71 per cent of children who commenced the group work going on to complete the programme in full, representing 49 per cent of all children referred to Family SMILES.

The majority of parents who completed the programme were women. Only 14 per cent of adult referrals to Family SMILES were men; this is further reflected in the low number of fathers and male carers who completed the programme and took part in the evaluation (around 10 per cent). This may reflect women being more likely to be the main carer of children and therefore more likely to be referred to programmes such as Family SMILES. Inclusion criteria for the programme required the parent to have a formally diagnosed mental health difficulty and to be receiving ongoing professional support. The support could be offered by a GP as well as the mental health team. The inclusion criteria were not always adhered to, however: some parents had not had a formal diagnosis or were not receiving mental health support services. The decision to offer the Family Smiles service would have been taken on the basis that it was in the child’s best interests, although this introduces a risk that the programme does not have the necessary links with adult services should the parent’s mental health deteriorate.

An overview of the completion of the evaluation measures at the different time points is provided in Table 1. Three measures represented the child’s perspective: the adapted Rosenberg self-esteem scale; the children’s evaluation wheel and the Strengths and Difficulties Questionnaire. The evaluation wheel was administered at programme start (Time 1) and completion (Time 2) only; both of the other measures were also, in addition, completed at follow up (Time 3). There is a complexity with the SDQ that although a large majority of the forms were completed by the children themselves, in one in five cases the measures were completed by their parents: of the 199 SDQs analysed for Time 1 and Time 2, 159 were completed by children and 40 by parents or carers. The analysis required that the same person completed the measure at Time 1 and Time 2, so in the few cases where the SDQ was completed by a different person across the time points, those measures were not included in the analysis.


There was significant attrition between the end-of-programme evaluation and the follow up six months later, as practitioners sometimes found the latter hard to achieve, for reasons that included changes in the families’ circumstances. Of the three child-focused measures, only the SDQ was used in the comparison group. The low numbers at Time 0 (following referral and during assessment for the programme) reflect the late introduction of the waiting list comparison group.

Just over half of parents who started the programme (52%, n=188) completed a CAPI at Time 1 and a large majority of these (89 per cent) were valid for analysis, as judged by the validity scales incorporated into the measure. A third of parents who completed the programme in full (31 per cent) completed a valid CAPI at both Time 1 and Time 2. As with the children, there was significant attrition between end of programme and follow up; occasionally this was because the practitioners were concerned for the parent’s mental health and did not think it would be helpful for them to complete the measure at that point. The parents’ evaluation wheel was the second measure to be included at Time 0 for the comparison group.

To summarise, the pre-/post-evaluation sample (Time 1 and Time 2 measures) therefore represents 31 per cent of parents who completed the programme in full (based on the CAPI) and 60 per cent of children who completed the programme in full (based on the SDQ).

The profile of the evaluation sample was very similar to that of all children who accessed Family SMILES in terms of gender distribution and age range (most children fell into the 10–11 years age category).

Appendix 5 outlines in full the numbers of completed measures for children and parents at each stage of the evaluation and their key characteristics.


Table 1: Overview of number of completed outcome measures

Outcome Tool Perspective Comparison Group (T0 & T1)

Pre- /Post- programme (T1 & T2)

6-month follow up (T1, T2 & T3)

Increased self-esteem among children

Adapted Rosenberg Self- Esteem Scale

Child N/A 209 69

Reduced emotional and behavioural difficulties among children

Strengths and Difficulties Questionnaire (SDQ)

HoNOSCA

Child/Parent

Practitioner 31

N/A

199

164

65

N/A

Children have an improved ability to process thoughts and feelings

Children’s evaluation wheel

Child N/A 230 N/A

Improved protective parenting/improved safeguarding of children

Child Abuse Potential Inventory (CAPI)

Parent’s evaluation wheel

Parent N/A

22

74 (59 of which are valid*)

74

22 (18 of which are valid*)

N/A

* Measures were deemed invalid if a significant number of responses were missing or the validity scales contained within CAPI indicated the parent ‘faking good’ or ‘faking bad’ responses.

Qualitative interviews

This report also draws on qualitative data from interviews with children and parents who completed the programme, practitioners who delivered the programme and professionals who referred parents and children to Family SMILES.

The aims of the qualitative interviews were to:

• understand any changes for children and parents after attending Family SMILES

• identify the facilitators and barriers to service users achieving positive outcomes

• understand their experience of the programme.


Sample of children and parents for qualitative interviews

A total of 15 interviews with children and 16 interviews with parents/partners/carers were undertaken. Children and parents were selected through purposive sampling, using criteria that reflected key differences in the study population relevant to the study’s objectives, rather than trying to ensure the sample was statistically representative.

Our key criterion in choosing children or parents to be interviewed was whether they perceived an improvement (on one or more of the standardised measures) or not. In addition, we included children and parents who had refused consent to complete evaluation measures, and the parents/carers who did not have the mental health problem, to understand their experiences of the programme.

While we attempted to get equal numbers across categories this was not possible (see Table 2) due to difficulties in contacting children for whom there had not been an improvement (according to one or more standardised measures) or parents who had not agreed to complete the evaluation measures.

Table 2: Numbers of children and parents interviewed based on pre-/post-change

Pre-/Post-change reported on measures Children Parents/carer

Improved 9 3

Same/Got worse 3 2

Refused consent to complete measures 3 7

Partner/Carer (who did not have the mental health problem)

0 4*

Total 15 16

*Includes two parents/carers whose partners also took part in interviews.

The type of parental mental health problem was also considered in selecting a diverse sample. The range of conditions experienced by the parents is outlined in Table 3.

Table 3: Nature of mental health problem

Diagnosis Number

Bipolar 3

Personality disorder 2

Depression/Anxiety 6

Post Natal Depression 1

Obsessive Compulsive Disorder 1

No specific diagnosis 1


The children’s ages and geographic distribution of the families can be found in Tables 4 and 5. While the sample included children of various ages, there was a slight bias towards older children (age 11 or above).

Table 4: Age and gender of children who were interviewed

10 years and under 11 years and over* Total

Girls 3 6 9

Boys 1 5 6

Total 4 11 15

* Children aged over 13 may have been younger at time of attending Family SMILES

Table 5: Geographic distribution of children and parents interviewed

Location Number of children

Number of Parents/Carers

Number of families

Belfast 4 4 3

Bristol 1 1 1

Middlesbrough 3 3 3

Prestatyn 4 3 2

Southampton 3 5 5

Total 15 16 14

Most interviews were done individually with each child and parent and lasted approximately 45 minutes, although two did take place together. The interview schedules are attached in Appendix 2.

Sample of referrers and practitioners for qualitative interviews

Ten interviews with referrers took place, which are included in this final report.

Referrers were also selected using purposive sampling. The key criterion for sampling referrers was to ensure that they represented the breadth of referring agencies (see Table 6) to the programme, and that they represented the different locations across the UK.

Table 6: Types of referring agencies included in the sample

Referring agency Number of professionals

Adult Mental Health Services 2

Schools 2

Local Authority Children’s Services 4

Other voluntary sector organisations 2

Total 10


Referrers from the following geographic locations were included in the sample: Belfast, Bristol, Grimsby, Manchester, Middlesbrough and Prestatyn. All referrer interviews took place over the phone and lasted approximately 25 minutes. The interview schedule can be found in Appendix 2.

Nine interviews with NSPCC practitioners took place; practitioners were sampled to represent the views from a geographic spread of NSPCC service centres. Practitioners from the following service centres were included in the sample: Belfast, Bristol, Grimsby, Manchester, Middlesbrough, Prestatyn and Southampton.

These interviews also took place over the phone and took around 50 minutes. The interview schedule used with practitioners is attached in Appendix 2.

Ethics

The key ethical considerations that influenced the evaluation included:

• service users could give informed consent

• confidentiality and its limits were understood by participants

• they were aware of their option to withdraw from the evaluation or any aspect of it

• the principle of no harm to participants as a result of the evaluation was kept in mind while explaining measures or conducting the interviews

• participants had access to advice or support related to the evaluation

• participants, practitioners and the evaluation officers had access to debrief sessions to process any concerns raised through the evaluation.

A note on the ethical considerations is attached in Appendix 3.

Prior to the study commencing it was approved by the NSPCC’s Research Ethics Committee (REC). The REC includes external experts and senior NSPCC staff members. This ethics governance procedure is in line with the requirements of the Economic and Social Research Council and Government Social Research Unit Research Ethics Frameworks.


Analysis

The responses to the evaluation measures were analysed using a range of statistical tests. Statistical significance indicates the extent to which scores at one point in time are different from those obtained at another point in time. This report uses the convention that a change is considered statistically significant if there is less than a five per cent chance of it happening randomly, i.e. simply by chance. It is also useful to explore whether there are clinically relevant changes in scores on standardised measures, such as the SDQ, to aid our understanding of whether the difference found is actually meaningful in the ‘real world’. For instance, a child’s very high score on the SDQ at initial assessment may indicate their feelings or behaviour are causing concern, and that their needs are above the clinical threshold, requiring additional support or treatment. However, if at the next assessment their score drops substantially (below the clinical threshold) this would suggest significant improvement. This real-world difference is called a clinically significant difference, which is helpful for interpreting results in addition to statistical significance. Further information about the analysis of measures is outlined in Appendix 6.

The qualitative data from the interviews was analysed using a Framework ‘case and theme’ approach (Ritchie and Lewis, 2003). The list of themes developed is attached in Appendix 6.

Limitations of the research

There were also a number of challenges associated with collecting CAPI data from parents. Some practitioners expressed concerns about offering the CAPI due to the length of the questionnaire and the language it used, which they felt might act as a barrier to establishing positive relationships with parents. These challenges contributed to slightly lower numbers of completed and valid questionnaires at Time 1 and Time 2 than anticipated.

An additional issue was that the number of valid evaluation returns from parents (CAPI) at Time 3 (six-month follow up) was low. Challenges in collecting data six months after programme completion also related to fluctuations in parents’ mental health problems and changes to family circumstances once their engagement with the NSPCC had ended, adding further complexity to follow up with these families. It has therefore only been possible to undertake descriptive analysis of this data rather than applying statistical tests to understand trends in sustained changes for parents.


Chapter 2: Outcomes for children and young people This chapter will explore the changes experienced by children who went through the Family SMILES programme. In addition to pre- and post-intervention data as well as comparison group data, it provides an insight into what these changes felt like for children by drawing on qualitative data from interviews. It will also explore facilitators and barriers to outcomes being achieved for children and young people.

2.1 Change in emotional and behavioural problems

2.1.1 Parents’ and children’s perspectives: SDQ

The decrease in the children’s mean SDQ score from beginning to end of the programme, from 16.5 to 15.5 was small but highly statistically significant nevertheless (p=0.001) – meaning that it is not likely to have happened by chance. It indicates an overall improvement in children’s emotional wellbeing and behaviour, albeit small.

Change in level of difficulties experienced by children

Any score falling within the ‘high’ or ‘very high’ scoring bands was deemed to indicate a clinical level of difficulty. By this measure, 44 per cent of children started Family SMILES with a clinical level of emotional and behavioural difficulties, but by the end of the programme, just over one third of them (37%) had moved into the non-clinical range of need. This reduction in clinical need was not only clinically significant but also statistically significant (p = 0.046). Figure 3 below shows the proportional movement in children who started Family SMILES with a clinical level of need and ended with a non-clinical level of need at the end of the programme, and vice versa.

Figure 3: Proportion of children moving out of clinical and non-clinical difficulty by the end of Family SMILES

Non-clinical T2 Clinical T2

0% 20% 40% 60% 80% 100%

Clinical T1 (n=87)

Non-clinical T1 (n=112)


Comparison group data

Thirty-one children completed the SDQ at the point of referral to Family SMILES (Time 0), and again just before they started the programme (Time 1). There was a very small increase in mean SDQ score from Time 0 to Time 1, from 19.55 to 19.68, so remaining within the ‘high’ score band and above the threshold of clinical need for emotional and behavioural difficulties. The proportion of children whose score improved on the SDQ was smaller for the comparison group (31 per cent of children had an SDQ score that improved while waiting for the service), than for the intervention group (where 44 per cent saw an improvement in their SDQ score after programme completion. This suggests little improvement for children as they wait to start the programme, and for this cohort of children, their level of difficulty remained at a level of clinical concern during this time.

With regard to movement out of the clinical concern range, there was clinical improvement for 14 per cent of children who were waiting to receive the service, compared to 37 per cent of children who completed Family SMILES. Figure 4 shows the proportional movement in children from a clinical level of need at Time 0 to a non-clinical level of need at Time 1 and vice versa. This indicates greater clinical improvement for children who completed the programme compared to those in the comparison group.

Figure 4: Proportion of children moving out of clinical and non-clinical difficulty in the comparison group

Non-clinical T1 Clinical T1

0% 20% 40% 60% 80% 100%

Clinical T0 (n=21)

Non-clinical T0 (n=10)

It is worth noting, however, that a much larger proportion of children in the comparison group started with a clinical level of need compared to those in the intervention group. It is not clear what may have contributed to this difference between these two cohorts of children. While the gender balance of the comparison group and intervention group were fairly similar, there were proportionally fewer children aged 12 years and over in the comparison group, which may have affected SDQ scoring. A full breakdown of the characteristics of the evaluation sample can be found in Appendix 5. It is also possible that practitioners prioritised making evaluation visits to families where they were more concerned about the child, to try and ensure their


engagement in Family SMILES when space in the programme became available for them.

Change in SDQ subscales

The SDQ subscales provide an insight into where improvement occurred for children who completed Family SMILES. Figure 5 below compares the proportion of children experiencing clinical levels of difficulty for each of the SDQ subscales pre- and post-Family SMILES.

Figure 5: Clinical difficulty by SDQ subscale pre- and post-Family SMILES

T1 (% clinical) T2 (% clinical)

60

55

50

45

40

35

30

25

20

15

10

5

0

Condu

ct

Emotion

al

Hypera

ctivit

y

Peer

prob

lems

Pro

socia

l

With the exception of peer problems, there was a decrease in the proportion of children with a clinical level of difficulty across the subscales by the end of Family SMILES. The change in scores for conduct and hyperactivity subscales were statistically significant (p=0.004 and p=0.06, respectively), indicating a clear improvement in these two areas for children who completed the programme. However, this improvement was not evident when it came to peer problems, with most children (over 55 per cent) remaining at a clinical level of difficulty by the end of Family SMILES. This finding is explored in more detail in Chapter 5 of this report.


Parents’ and children’s perspectives on emotional and behavioural problems

Since SDQs were completed either by children or by their parent/carer, it is possible to explore the similarities and differences in their perceptions of the level of difficulty facing the child. In particular, looking at the subscales data provides a valuable insight into the ways in which children and parents perceived the child’s strengths and difficulties. For example, changes in parents’ SDQ scores for emotional difficulty between the beginning and end of the programme were statistically significant, but they were not when the child and parent scores were combined. This suggests that parents did perceive an improvement to children’s emotional difficulties at the end of the programme although this perception was not shared by children themselves. Further, parents’ SDQ scores crossed the threshold from a level of clinical to non-clinical concern by the end of the programme. This therefore suggests a clinically significant change in the way parents perceived their children’s emotional difficulties.

Although changes in peer problems were not statistically significant for either parent- or child-completed SDQs, this was perceived as a more serious problem by children than it was by parents. Children perceived that peer problems remained at a clinical level at the end of the programme, while parents tended to report a non-clinical level of difficulty in this area for their children. These findings highlight how perceptions of levels of difficulty may vary between parents and children.

2.1.3 Practitioners’ perspective

The mean score for the HoNOSCA (n =164) decreased from 7.6 at the beginning of the work to 6.1 at Time 2, a small change that was nevertheless statistically significant (p<0.00) reflecting a decrease in children’s difficulties – as perceived by the practitioners. Due to the absence of any cut-off points for the HoNOSCA, it is not possible to determine the clinical significance of this change.

2.1.4 Examples of change in children’s emotional and behavioural problems

The qualitative interviews highlighted changes in children’s behaviour. One change that was noted was the way in which children behaved among their peers. Some children, who had previously struggled to interact with others, were more able to play appropriately and build friendships with other children, as in this example:


“children were a bit scared of him [child] in a way… And he has

been able to make friends better because of it [family smiles].

he’s been able to socialise a lot better; he’s been able to

concentrate on his work more.”

Referrer, primary school

In addition to changes in the way children interacted with one another, parents also perceived an improvement in their children’s behaviour at home. Where some children had previously been uncertain about how to respond to their parents’ mental health problems, they were more helpful and positive in their attitude at the end of the programme. Parents described how children displayed more empathy and showed gestures of support to their parents when they were feeling unwell:

“if i was down or if i was crying, he would…[say]: ‘are you alright

mum?’ ‘make you a wee cup of tea?’ doing all those wee things

and he’d come over and give me a hug more and it was just, he

didn’t look worried and that was really great”

Parent, depression/anxiety

“if i got a bit down or i’d say to her, ‘oh, would you make me a

coffee please?’ we’d have like, “Argh!” and … shouting and that

but after she’d been to quite a few meetings i’d say to her, ‘oh

would you make me a coffee please?’ ‘yeah sure,’ and it was

like: hang on is this my daughter?”


At the other end of the spectrum were those children who had been identified as playing too much of an adult role previously and who had been preoccupied with providing care to their parent. For some of these children it was noted that they began to relax and behave more like children after attending Family SMILES:

“he sort of ‘de-roled’ as an adult and became more of a child

and interacted really well with the other children.”

Family SMILES practitioner


“i think they got a bit worse. however, in a way that was a good

thing because it just meant that stuff was coming out for them.

so it’s been really important to me that no matter how kind of

unwell i’ve been, that they don’t have to stop being children, and

that they don’t take on too much responsibility… to see them

being a little grumpy or naughty was kind of nice, because for a

long time they had been really, really unnaturally good because

they didn’t want to cause me any stress”

Parent, Obsessive Compulsive Disorder

In addition to changes in children’ behaviour, respondents also described changes in children’s emotional difficulties: in particular, how children felt less anxious and more relieved and reassured after attending Family SMILES:

“she used to struggle to get to sleep, and i think that was mainly

worry, and she used to also get quite an upset stomach, i think

again through worry, and she hardly has those problems now.

yeah, it was sort of mental things that were manifesting as

physical problems.”

Parent, bipolar

“i think it’s a bit of a relief in her, to be honest. she’s generally

more happy and outgoing and more comfortable with things.

yeah, i think a bit of relief.”

Parent, bipolar

Children who had previously struggled with the emotional stress of living with a parent with mental health problems were able to deal with these emotions differently after attending the programme; these changes manifested in the way children felt emotionally as well as physically.


2.2 Change in children and young people’s self esteemThe Rosenberg Self-esteem (adapted) questionnaire (n=209) was completed by children and used to measure changes in their self-esteem over the course of the Family SMILES programme. This data shows the mean score increasing from 18.5 at the start of the work to 19.5 at the end (see Appendix 6 Table III). This increase is statistically significant (p= 0.001), indicating an increase in children and young people’s self-esteem. Since this measure has been adapted, there are no cut-off points defining clinical levels of high or low self-esteem. It is therefore not possible to deduce from the data whether the scores reflect a clinically high or low level of self-esteem.

2.2.1 Examples of change in self-esteem

The improvement in children’s self-esteem was described during interviews. Parents in particular, spoke about changes in how their children were feeling about themselves and the way this was reflected in their participation in a range of new activities as well as displaying greater levels of confidence within their peer groups, as in these examples:

“there were a few kids in next door’s house and they were all

dancing and carrying on. usually [child] would sit in the corner,

but no, she wasn’t, she was at the front; ‘ahh, look at me!’ next

door just couldn’t believe it – this was like three weeks ago – and

she would never have done that before.”

Parent, no specific diagnosis

“[child’s] confidence has gone through the roof [compared] to

what she used to be like… she’s always in the school plays

and she’s pupil of the year and pupil of the week all of the time.

there’s loads of stuff in there on the wall, our ‘Wall of fame’

i call it.”

Carer, ex-partner with depression/anxiety

NSPCC practitioners also reflected on transformations in children over the course of Family SMILES, describing children who had initially felt unable to contribute to discussions developing their confidence and self-esteem, and participating enthusiastically in all group activities by the end of the programme.


2.3 Change in children’s ability to process their thoughts and feelings A total of 230 children’s evaluation wheels were completed at the beginning and end of Family SMILES. The evaluation wheel captured children’s ability to process their thoughts and feelings. The data are presented below in Table 7; possible scores for each dimension on the wheel ranged from 0 to 5.

Table 7: Data from children’s evaluation wheel (n= 230)

Dimension Pre- programme

Post- programme

Statistically significant

I can talk to someone if I’m worried about my parent’s health.

3.6 4.1 Yes

I can talk to my parent about how their mental health affects me.

3.1 3.5 Yes

I can easily make friends. 3.8 3.9 No

I am able to have fun when I want to.

3.9 3.9 No

I feel supported by others around me.

4.0 4.1 Yes

The children’s evaluation wheel data indicates that:

• There was a statistically significant improvement for three criteria on the evaluation wheel. The two criteria that did not have a statistically significant improvement were feeling ‘able to make friends easily’, and ‘having fun’. Scores on these two criteria started at a relatively high level and remained there at the end of the programme.

• At the start of Family SMILES, children rated being able to talk about their worries regarding their parent’s mental health as the most problematic of the evaluation wheel criteria. The subsequent improvement is likely to reflect the change in children’s self-esteem, with children feeling more comfortable in themselves and able to express their thoughts and feelings at the end of the programme.


2.3.1 Changes in children’s ability to process and express their thoughts and feelings

During the qualitative interviews, children described how Family SMILES had helped them to deal with thoughts and feelings associated with their parent’s mental health needs. By being reassured about their situations at home, children were better able to understand and process their thoughts and feelings appropriately:

“Well, i think the group has just been sort of really reassuring

to help us get through all the doctors coming in, and my mum

having to go out to the hospital every now and again to get tests

and stuff like that. they were just reassuring; sort of helping us

out with that, getting through it.”

Boy, 12 years

Children also reported that not only did they feel more able to process their experiences internally, but also to express their worries, thoughts and feelings to trusted individuals. This gave them a sense that they could share the burden of their emotions with others, helping them to continue to process and deal with their thoughts and feelings appropriately:

“i never used to express myself, …whenever i felt down i used

to go and sit up in my room and i used to not want to talk to

anyone; i used to get all worked up and bottle it up inside me.

But then when i had been to that group for eight weeks [family

smiiles] i just couldn’t end up holding it in and maybe that was a

better way of showing that i’m hurting…”

Girl, 14 years

“it gave me a chance to get everything out of my mind so i could

actually say it to someone, so someone else knew what i felt.”

Boy, 12 years

These changes reflect the improvement in children feeling able to talk about worries associated with their parents’ mental health, captured by the children’s evaluation wheels.


In summary

The key findings around outcomes for children and young people indicate that:

• Emotional and behavioural problems decreased among children by the end of Family SMILES; in particular, hyperactivity was reduced and conduct had improved. Parents also perceived a significant improvement in children’s emotional difficulties; however, children did not share their perspective on this improvement.

• There was a reduction in the proportion of children with difficulties at the highest levels, which indicates clinical significance.

• There was greater improvement (both in terms of movement to a lower scoring band on the SDQ and clinical improvement) for children who completed the programme compared to children who received the assessment only in the comparison group. This suggests that the improvement experienced by children who completed Family SMILES, can be attributed to the programme.

• There was an improvement in children’s self-esteem by the end of Family SMILES.

• There was an overall increase in children’s confidence around talking about their worries related to their parents’ mental health problems and an improvement in their sense of support from others.

2.4 Factors affecting outcomes for children and young peopleThe qualitative interviews highlighted key barriers and facilitators to the achievement of the desired outcomes for children.

2.4.1 Facilitators

Meeting other children facing similar difficulties

The children’s group was described during qualitative interviews as a support network for children engaged in Family SMILES. It provided them with the opportunity both to share and listen to the experiences of others and to seek one-to-one help from the NSPCC worker when they felt this was needed.


Children described how they found it easier to speak in front of other children with similar experiences to them, rather than members of their own families. Parents also perceived that one particular strength of the programme was the confidential space it provided for children to express themselves:

“it was easy because you didn’t know them as well… you knew

you weren’t hurting them by saying all this because obviously

they’re not part of your family or nothing to do with you.”

Girl, 14 years

“i think the most important thing for them was that they met other

children who were in a similar situation to them. so they didn’t

feel they were the only ones and that they were able to speak

openly about things.”


Being in a peer group also helped to reduce children’s sense of isolation and enabled them to realise that there were other children who, like them, were also living with parents experiencing mental health problems:

“instead of maybe thinking he was maybe the only one,

compared with his other friends in school and stuff. i think it was

really good for him in the sense that he got down and seen ‘it’s

not just me.’”


Enabling children to share their experiences with peers who understood their situation had the additional benefit of providing children with the opportunity to build new friendships:

“i think me and a girl called [name] really bonded. like on the first

day… i had the pink chair and [name] was on the purple chair

right next to me. And if we got stuck we’d ask each other and i

think that really helped me, to know that i’d made a new friend.”

Girl, 10 years


“it was a confidence booster to them [children] because the lads

are quite quiet and shy, where they seemed to make some good

friends in the group. so, again, i think it was good there was a

mixture of children from different schools.”

Parent, bipolar

Building new friendships contributed to children’s enjoyment of the group work. Some children who struggled socially before coming to the programme were able to build friendships with children who shared a mutual understanding of living with parental mental health problems.

Learning about mental health

The approach taken by NSPCC practitioners running Family SMILES enabled children to grasp complex concepts around mental health that they had previously not been able to understand. Children and parents reflected how sessions exploring different types of mental health problems had been useful, as they allowed children to talk generally about mental illness without naming their parents’ problems:

“i didn’t really understand it before, it was really confusing, but

now i think i understand a lot better and it’s not as scary as you

think it is.”

Girl, 12 years

“they have a good understanding of mum’s condition now and

mental health disorders, it’s something i’ve tried explaining to

them [children] but the nspcc do it in a different way which is

obviously far easier for them to understand.”

Carer, bipolar parent (daughter)

Children were able to apply this new learning to making sense of their parents’ specific mental health conditions. This helped them develop a sense of empathy for their parents’ situation, a greater understanding of their behaviour and of their needs:

“i found things helpful, like what sort of illnesses my mum had,

and it’s just that i’ve been wondering what sort of things she has

got… now i know what my mum’s got, it’s sort of like at least my

mum has told me instead of kept it a secret for her life”

Girl, 10 years


“Before, i kept a lot from her, she just knew that sometimes i was

a bit hyper, sometimes i was a bit sad, and she didn’t really know

why or understand it… the bits that they did in the groups, the

sessions that they had, she’s really informed now, she knows

what she should know.”

Parent, bipolar

Believing they are not to blame for parental mental health problems

Meeting other children with similar experiences and gaining new knowledge about mental health not only provided children with a greater insight into their parents’ conditions but reassured children that they were not to blame for what their parents were experiencing:

“i always told her it wasn’t her fault when i was poorly and i

locked myself away, that it wasn’t because i didn’t love her. But i

think she just thought, ‘you’re just saying that’. so i think being in

a group with lots of other people, she believes that now and her

confidence has grown each week throughout smiles.”


Believing that their parent’s mental health problems were not their fault provided children with a sense of relief, as they were able to recognise that this was not something for which they should blame themselves or feel responsible:

“i think the difference was that she didn’t believe me and now

she does. now she knows that i don’t go into myself because

she’s getting on my nerves, i go into myself because that’s what

i do, that’s part of my illness… and because she knows it’s not

her fault anymore i think a massive weight’s been lifted off her,

because i think she felt so responsible for it before and now

she doesn’t.”


“We were told that the mental illnesses and our parents getting ill

wasn’t our fault… so, it was reassuring because i thought many

times when mum used to go into hospital, i thought it was my

fault; but it wasn’t.”

Boy, 13 years


Child-centred model of service

The gap in provision of services for whole families where there are parental mental health problems was frequently mentioned by referrers to the service during qualitative interviews. In a context in which children’s feelings and wishes could be overlooked, the opportunity that Family SMILES provides to children to have their voices heard was therefore perceived as an important facilitator of change for children:

“With adult mental health… you’ve got a lot of professionals

working around them… the temptation is, you feel it’s almost

whoever is chatting louder gets the attention a lot of the time, and

it probably isn’t the children.”

Referrer, Voluntary Sector Organisation

“obviously everyone knew, the doctors that were involved and

the social workers that were involved, knew what was going on.

But just generally having someone that could listen to my point

of view of it and how i felt with everything… and they could talk

to the social workers and get some things changed; it did help

a lot.”

Boy, 12 years

The Family SMILES model, which includes one-to-one work with the parent, as well as group work for children, also ensures that the burden of coping with the parent’s mental health is not placed on the child:

“if they’re doing the work and the parent isn’t, then i can imagine,

you know, they believe more that they’re the responsible ones,

because they’re the ones that are going off and doing it, whilst

mum’s not”

Referrer, local authority social worker

The support planning element of the programme that brings children and parents together at the end of Family SMILES further enables the voice of the child to remain at the centre of the work and for parents to understand the needs and wishes of their child.

Support planning provided children with the space to discuss what would happen in a crisis situation and to have their views on this heard by their parents. It presented the opportunity to address all the child’s anxieties about what would happen if their parent was unwell. Bringing parents and children together helped them to communicate


and open discussions about mental health in an age-appropriate way. It also provided the child with something they could refer to once they had finished the programme.

Creative activities and a positive approach

Children described the creative activities they took part in as enjoyable, helping them to improve their understanding of mental health problems. These included using ceramics, photography and painting as well as watching DVDs and creating hats or masks to highlight how people hide their feelings. Learning coping and relaxation techniques for dealing with stressful periods at home were also valued by children, and during the qualitative interviews children told how they continued to use these techniques at home after finishing the programme.

Children also valued activities that highlighted to them how widespread mental health problems are. One example of such an activity involved looking at photos of different celebrities and trying to identify which ones might have mental health problems. Children often felt reassured to discover how many other people also struggle with their mental health, addressing feelings of isolation and helping children to realise it should not be a reason for anyone to undervalue themselves:

“she went from not really wanting to talk about it or relating it to

her situation to she was googling different celebrities who had

mental illnesses; really interested in the ones that had bipolar;

brought them in and shared it with the group.”


“on one day we had all these celebrities and we had to guess

what one had a [mental] illness and what one didn’t. And at

the end of it, all of them had a [mental] illness… it helped me

understand that if people look young and healthy, it doesn’t

mean that they don’t have a mental illness, anybody can have a

mental illness.”

Girl, 10 years

In terms of how practitioners approached delivery of the programme, the importance of being flexible in how sessions were run was highlighted, for example being able to address issues such as bullying in a creative and more in-depth way when it was apparent that it was a problem affecting children in the group. Providing positive feedback throughout the programme was also identified as important for helping children and young people to achieve positive outcomes:


“[child] has quite challenging behaviour and is always getting

negative feedback; it was very rare that there was anything

positive coming back from school. he’s been banned from youth

clubs and things. But when he was part of smiles and with

[practitioner], it was like really glowing reports; lots of positive

feedback. And, you know, he really kind of thrived on that”.

Referrer, local authority social worker

2.5.1 Barriers

Children’s understanding about mental health before Family SMILES

Children who began the programme with less knowledge of their parent’s mental health problems than practitioners had initially anticipated sometimes ended up confused or angry with their parent. They felt that their parent had been withholding information from them, which exacerbated problems in their relationship. Likewise, children who struggled to grasp the range of mental health problems that exist sometimes felt confused or frustrated after listening to other children talk about mental health conditions that were different to those experienced by their parent:

“some children have been really angry with parents and saying

‘you didn’t tell me you had a mental illness’ and it’s almost like it’s

only just sort of dawned on them when we’ve started breaking it

down more in-depth about what it is that it’s actually starting to

ring true with them i think.”


“i said, ‘What’s wrong’, ‘nothing’. And he kept looking out the

window and i thought, ‘it’s really weird’. And then i started to

drive away and he said, ‘Why did you spend all our money?’

And i went, ‘What? i didn’t spend all the money’… it was another

wee girl in the group, her mummy was really bad with it and her

mummy spent all their money. so [child] was picking up and

thinking… that i was doing all this.”

Parent, postnatal depression

While some children and young people struggled to grasp key concepts during the group work because they had little initial knowledge about mental health, there were others whose views on mental health and mental health practitioners were so engrained that the programme did not change their views:


“they [local authority] want them [children] to see a psychiatrist

but they’re not willing to see one because i see one and they

said ‘mummy it’s not good to see one of those people. they just

brainwash you’”


The extent to which children sometimes struggled to process new information about mental health therefore had the potential to act as a barrier to positive outcomes being achieved.

Challenging behaviour in the group

The dynamics within children’s groups sometimes acted as a barrier, particularly when some children displayed challenging behaviour. It was not always easy to manage this, as these children had a tendency to get over-excited, to have trouble concentrating, or to be disruptive. This had the potential to lead either to children having to drop out of the programme early or to minimise positive outcomes for other children in the group:

“We did give him another chance but he was just not willing to

accept that he did anything wrong. And it was like well actually

you can’t come back because he was running out of the building

and all sorts”


Children also reflected on the frustration of being part of a group where the behaviour of others prevented them from getting the most out of the programme:

“some weeks it was like if the boys weren’t like hyper or like

mucking about then it was fine but then if they was like really

hyper they would just be laughing at everything everyone was

saying or just used to mess about and you was just like, yeah i’m

not going to say nothing. so it was like more uncomfortable when

they were really hyper or excited”

Girl, 15 years

“i think they did understand things, but they took it all in as a

joke. like they weren’t very serious in the serious things we were

talking about.”

Girl, 10 years


During qualitative interviews, practitioners described the importance of taking a united approach with their co-workers to deal with challenging behaviour to ensure the smooth running of the groups.

Parental concerns around children learning about mental health

Some parents who had agreed to their children taking part in Family SMILES remained unconvinced that their child should be learning about mental health at all. Parents who felt that their mental health problems were issues they had already dealt with that did not affect their children, did not see the benefit:

“Well to be honest, i didn’t agree with it at first about her even

going and finding out what’s wrong with her mother. she’s 10

– well she was 9 at the time – and i thought it was maybe too

much too soon.”

Carer, ex-partner has depression/anxiety

“As they [nspcc] heard ‘postnatal depression’, that’s what

they stuck on me, and i didn’t like that because that happened

whenever my daughter was wee. And yes, i was at a point now

where i’d got over it and i could talk about it and understand

why that was happening, but to me it was something that [child]

didn’t really need.”

Parent, postnatal depression

Parents who felt it was inappropriate for their child to be learning about mental health did not regard it as an important subject of conversation, thus limiting further discussions at home, which was a barrier to achieving outcomes for children.

Children using new information about mental health inappropriately

Apart from being uncomfortable with their children gaining new knowledge about mental health, some parents also expressed concern that this newfound knowledge provided the children with a new excuse to misbehave at home:


“she ended up knowing a little bit too much, i felt, because she

used a little bit of it against me at one point…. one time she

turned round and used it as a weapon against me because she’d

been naughty and said ‘if i don’t do it [family smiles], it won’t

work’ because she’d heard it said in the meeting ‘that we all have

to do it as a family.’”


“now he does understand a little bit more but it’s like he sees

it as an opportunity to play up. so if i tell him i’m having a bad

day or i’m stressed and just give me a bit of time then they’re

constantly there playing up, and that doesn’t help.”

Parent, personality disorder

In some instances, parents perceived that children used the new information about mental health to misbehave at home. Such cases demonstrate this could be a barrier to children benefitting from the programme, since the only change parents recognised was the deterioration in the behaviour of their children.

Coping with the group ending

During qualitative interviews, the length of the programme and the difficulty some children experienced accepting that the group work had ended also emerged as a barrier to outcomes being realised for children. This was especially the case where children had benefitted from meeting other children similar to themselves and building friendships in the group:

“i felt a bit sad ‘cos it’s a shame that i might not bump into them

again, you know. it’s tricky saying goodbye, isn’t it, really?”

Girl, 10 years

While many children were able to go on to join an NSPCC participation group at the end of the programme, or were referred to other support services in the area, it was suggested that having continued support in place following Family SMILES would help ensure continued positive outcomes for more children.


Support planning not appropriate for the family situation

The joint session at the end of Family SMILES provided an opportunity for parents and children to come together to draw up a support plan. While this presented a valuable opportunity for many families, for others this piece of work was seen to be more detrimental, especially where it raised anxiety levels within the family about a potential crisis:

“you know, no matter how much you explain that it’s just in the

event of an emergency they always sort of think ‘well, my dad

would be there or my mum would be there, why are you saying

this? so are they not going to be there?’ And i found that quite

worrying, really.”


“i think the plan was a little bit generic, so some of the stuff

didn’t really apply to us. i mean, because i’m really quite well

now and so i think some of it might have worried the kids a little

bit. Because that kind of thing is like ‘think of the worst case

scenario’ and stuff like that. And i don’t really think that they

[children] wanted to have to think about that stuff again.”


Other parents felt that it was a negative way to end the programme as there was already sufficient support for the children at home:

“i suppose it would have been nice to have…a consolidation

session where we could all sit down to discuss what our

understanding, coming out of the programme and stuff… hopes

for our family or future of our family…something with a nice

positive direction. i think like that would have been really handy.”


The way in which the programme was brought to an end was felt to be important by service users and practitioners alike. Finding a way to end the work as positively as possible for the children was perceived as key for ensuring positive outcomes.


In summary

The key facilitators to help achieve positive change for children were:

• meeting other children facing similar difficulties

• improved knowledge and understanding about mental illness

• believing that they were not to blame for parental mental health problems

• the child-centred model of the programme

• creative activities.

And the key barriers to achieving those outcomes were:

• children’s limited understanding of mental health before Family SMILES

• challenging behaviour in the group

• parental concerns about children learning about mental health

• children using new information about mental health inappropriately

• difficulty coping with the group ending

• support planning not appropriate for particular family situations.


Chapter 3: Outcomes for parentsIn addition to the group work with children, an integral part of the Family SMILES model is the individual work with the parent/carer with mental health problems. The aim of this work is to reduce parenting behaviours that contribute to the vulnerability and risk for the child and to enhance the parent-child relationship.

This chapter evaluates the extent to which protective parenting behaviour changed due to Family SMILES, based on data from the parents’ evaluation wheels and the Child Abuse Potential Inventory (CAPI).

3.1 Parents’ perceptions about their parenting and impact of mental health problems The mean scores for the parents’ evaluation wheels at the start and end of the programme are shown below in Table 8.

Table 8: Average scores of parents who completed the parent evaluation wheel before and after the programme (n=74)

Dimension Pre- programme score

Post- programme score

Statistical significance

How much I think my child is affected by my behaviour

3.88 3.92 No ( p=0.40)

How confident I feel that I am doing the best I can for my child

3.53 4.03 Yes (p<0.001)

How supported I feel in taking care of my child

3.47 3.95 Yes (p<0.001)

How confident I feel in asking for help when I need it

3.30 3.77 Yes (p=0.004)

How much knowledge I have about children’s needs at different stages of their development

3.78 4.10 Yes (p=0.007)

The parents’ evaluation wheel scores reveal a general increase in confidence and understanding across all of the criteria, though there was little change in perceptions relating to how much their children were affected by their behaviour. However, it is worth noting that at Time 1 this dimension started with the highest score, suggesting that this was not a key area of concern for parents. The findings paint a picture of parents having greater insight into their children’s needs and into the impact of their mental health on their children at the end of Family SMILES.


The open questions in the evaluation wheel enabled parents to reflect on how they enjoyed building a relationship with the NSPCC practitioner and benefitting from their support, knowing that their children were learning about mental health, and learning about how their mental health affected their children and wider family. Parents also admitted finding it hard to confront past experiences and having to deal with difficult emotions. In reflecting on what they had learnt from the programme, parents felt they were better able to communicate effectively with their children, and to ask for help from others when they needed it.

Comparison group data

Twenty-two parents completed the parent evaluation wheel at Time 0 and at Time 1. The differences in mean scores for the comparison group and for the intervention group are displayed in Table 9.

Table 9: Differences in mean scores on the parent evaluation wheel for the comparison group and intervention group

Evaluation wheel statement Difference in mean scores between T0 & T1 (Comparison Group, n=22)

Difference in mean scores between T1 & T2 (Intervention Group, n=74)

Change greater for intervention group?

How much I think my child is affected by my behaviour

– 0.23 + 0.04 Yes

How confident I feel that I am doing the best I can for my child

+ 0.22 + 0.50 Yes


+ 0.09 + 0.48 Yes

How confident I feel in asking for help when I need it

+ 0.70 + 0.48 No

How much knowledge I have about children’s needs at different stages of their development

+ 0.17 + 0.32 Yes

The parents evaluation wheel data indicate a greater improvement for the intervention group than the comparison group across all criteria, with the exception of ‘confidence in asking for help when needed’. This suggests that positive outcomes at the end of the programme are likely to be attributed to Family SMILES since the same extent of change did not exist for parents in the comparison group.

The greater improvement in confidence to ask for help in the comparison group may be due in part to parents having built a trusting relationship with the practitioner during the assessment, which possibly also prompted them to reflect more on their existing support network and made them realise they had underestimated the full extent of the support available to them. Anyhow, it was the


lowest scoring dimension at Time 0 (mean score of 2.9), so it had the most scope for improvement over the course of the assessment and the programme itself. Nevertheless, it is still not clear why a similar amount of change was not achieved through the Family SMILES programme.

Due to the small size of the comparison group, and the absence of standardisation of the evaluation wheel, this data should be interpreted with some caution.

3.2 Change in behaviours that contribute to riskThe previous section explored whether parents experienced any changes in their perception of the impact of their mental health on their children. This section now examines whether such improvements translated into perceived changes in their protective parenting behaviour. The key standardised measure that was used to assess this change was the CAPI.

The Child Abuse Potential Inventory (CAPI)

Seventy-four parents completed a CAPI at both Time 1 and Time 2, of which 59 (80 per cent) were valid for analysis. Table 10 shows the average CAPI scores at Time 1 and Time 2, and whether the changes were statistically significant or not.

Table 10: Average scores of parents who completed the CAPI before and after the programme (n=59)

Subscale Pre- programme

Post-programme

Statistically significant

Distress (cut-off 152) 190.3 169.7 Yes (p<0.001)

Rigidity (cut-off 30) 14.4 13.8 No

Unhappiness (cut-off 23) 38.4 30.6 Yes (p<0.001)

Problems with child and self (cut-off 11) 8.8 6.7 Yes (p=0.006)

Problems with family (cut-off 18) 18.3 15.7 No

Problems with others (cut-off 20) 17.1 16.6 No

Total score (cut-off 215) 287.2 253.4 Yes (p<0.001)

Ego strength scale 9.3 12.5 Yes (p<0.001)

Loneliness scale 12.01 11.4 Yes (p=0.03)

The average total CAPI score had decreased by the end of the programme. Though statistically significant, the score, being higher than 215, remained above the threshold for clinical concern. This indicates that although parents improved in terms of protective behaviours, the change was not clinically significant. And although there was clinical improvement for some parents, as a group they had not moved out of a level of clinical concern by the end of Family SMILES.


The CAPI subscales also provide a helpful insight into where changes took place:

• There were statistically significant reductions in parents’ distress and unhappiness scores. However, these changes did not represent clinical improvement, as the scores remained above the cut-off points for these subscales. This suggests that although parents did feel happier and less distressed at the end of the programme, they remained at a clinical level, possibly due to the ongoing mental health problems they faced.

• The ‘problems with child and self’ subscale also saw a statistically significant reduction – but the level of concern was below the non-clinical threshold to begin with.

• There were statistically significant improvements in the subscales of parents’ ‘ego strength’ and their level of loneliness. There are no cut-off points for these subscales, but the improvements indicate that parents may have felt more emotionally stable, and less isolated and alone, at the end of Family SMILES.

• There were no statistically significant changes in parents’ ‘rigidity’, ‘problems with family’ and ‘problems with others’. However, the scores for these subscales either remained or ended up below the threshold of clinical concern, suggesting that they were not areas of difficulty for parents at the end of Family SMILES.

3.3 Examples of changes for parents

Improved communication and more time spent with children

Both parents and children described during the interviews how they recognised an improvement in communication at home since attending Family SMILES. Children and parents spoke about the programme providing them with the language to talk confidently with each other about mental health in a way they had not been able to previously:

“she [mum] is a lot happier now. When she talks about it [her

mental health]… she can talk to me at my level now. she doesn’t

have to simplify it anymore because i know the basis of what’s

going on.”

Boy, 13 years


“i was able to talk to him more about it and i learnt that from the

smiles from my own sessions: it’s ok to discuss things with

your child and it’s ok to say ‘i’m not feeling well’ which before, i

would have said nothing – just ‘oh, mummy hurt her back today’

or ‘mummy tripped and fell’ or something. i’d come up with

these reasons for being in bed or if i’d been crying or whatever.

definitely, definitely made a big difference.”


Parents also developed an understanding of the benefits of spending more time together as a family and were able to recognise that this had previously been missing from their relationships with their children. As parents’ confidence in talking to their children increased, they were also more likely to set aside time to spend together:

“i remember going back one week and them being really proud

to tell me that they’d taken him out and they’d been assertive

and that was appropriately assertive in arranging an excursion, a

family day out and that had gone really, really well.”


“We’re spending a bit more time, taking the time to either talk

to each other or do something together, like play a game or

something else which i think before, we just took each other for

granted and we were both there but we weren’t really, unless we

went out somewhere together and did something together, we

weren’t really communicating. so i think that’s got a lot better.”

Parent, bipolar

Parental responsibility

Before attending Family SMILES, a number of parents felt that they were not always able to meet their children’s needs due to their mental health problems. After the programme, parents were more able to identify elements of their parenting that they could address, in spite of their ongoing mental health needs, and were willing to acknowledge and act on the responsibilities that they had as parents:

“there were quite a few sort of light bulb moments… she said,

‘oh yes, i understand it now, i understand why my daughter’s

feeling this way. i can understand that.” Because for her it was

always about ‘well, it’s me and it’s my mental illness’: she couldn’t

see the other side of the fence.”



“Whereas before it would always be the mental illness... she’

wasn’t making that the problem anymore, she was a bit more,

‘yes, this is what i need to do as a parent, this is what i need to

do as a mum’… so i thought that was a change as well, rather

than just always blaming: ‘well, that’s how i am, i can’t do it

because of my mental illness’. she took a shift in her thought

process there.”


Parents and children also gave examples of what these improvements in parenting looked like. Such changes included playing a more active role in supporting children at school and establishing routines at home for the whole family:

“she phoned me to tell me when the girl got her sAt results, so

she’s really been able to focus her time on her parenting and

praise and recognition of all of her daughter’s achievements.”


“We have like tea at a certain time and stuff like that, and my

mum takes the dogs out for a walk now instead of just leaving

them… i think that’s how she calms down like after a hard day.”

Boy, 14 years

Motivated to bring about further positive change

Parents felt better able to focus on the future and to think positively about life in general after attending Family SMILES. Practitioners also noted a change in the demeanour of some of the parents they had supported, in that they were looking forwards rather than feeling stuck and restricted by their mental ill health:

“her willingness to engage and be very open, her positivity, being

able to be future-focused rather than being stuck and her whole

demeanour much less lethargic.”



“Well, i’ve got a job interview tomorrow, so that’s a big change!

And it’s really sort of opened my eyes of the fact that [child] does

need her education and she needs me to be well and stable as

much as i can be and i want to provide a good life for her. Before

i met [practitioner], i was in a bit of a slump, i think, but i feel

generally just positive and that i want to do things.”

Parent, bipolar

Parents further demonstrated this change by their increased openness to accessing additional services and support. For example, during interviews they mentioned enrolling onto parenting courses and responding promptly to phone calls from the NSPCC, when previously they would have ignored contact from involved agencies.

In summary

The key findings around outcomes for parents are:

• The quantitative data collected from parents indicates that their confidence in parenting skills had improved at the end of the programme.

• While the data also indicated an improvement in their protective behaviours, this improvement was not clinically significant so this aspect of parenting remained of significant concern.

• There was greater improvement for parents who completed Family SMILES than for those in the comparison group on all criteria on the parent evaluation wheel, with the exception of their level of confidence in asking for help when they need it. It appears that the assessment process alone may support parents in improving confidence in this area.

• Due to the small size of the comparison group sample of parents and the absence of standardisation of the evaluation wheel, this data should be interpreted with some caution.

• Qualitative data highlighted examples of change for parents in their parenting skills and protective behaviours, including improved communication, being more proactive in their parenting, and feeling motivated to bring about more positive changes.


3.4 Factors affecting outcomes for parents

3.4.1 Facilitators

Opportunity to reflect on the impact of challenging life events

A key starting point in the work with parents was helping them to reflect on their lives and to identify events that had affected them, particularly their mental health. This was an important facilitator of change for some parents, who were supported by practitioners to confront difficult moments in their past and to make sense of the effects such events had on their lives:

“she made those links and was able to see that actually, ‘i wasn’t

responsible for me not developing those skills as a child, it was

my parents and i shouldn’t have taken on those roles when i

was young; that was really unfair.’ And she was able to place the

responsibility back with her parents, which enabled her to move

forward and be more available for her children now.”


Exploring past events enabled parents to find a way to let go of some difficult emotions and to recognise that they were not to blame for the impact of such events on their lives. This process also prompted parents to want to act now in order to prevent their children finding themselves in similar circumstances in the future:

“it was a good way of showing that you can’t change the past,

but we can certainly mould the future and how you need to be

doing that, because you don’t want your daughter to be having

the same thoughts and feelings that you’ve been, and that was

quite a powerful change.”


Parents acknowledged that delving into the past could be a painful experience, but where they were able to participate in such activities, they felt the benefit of doing so:

“he started trying to do a family tree with me which i had never

really done before and different experiences and things like

that and it was quite upsetting doing some of that work… they

helped me start to discuss things, well, that i hadn’t discussed

for such a long time.”



Focusing on parents’ strengths

During interviews, practitioners reflected on the benefits of focusing on parents’ strengths and using a solution-focused approach in order to bring about positive changes:

“our first part of the next session would be talking about her

reflection on the previous session and something that she’d

done, or things that we’d discussed about trying that she would

say, ‘oh well, it worked’ or ‘it started to be okay at this point’. so

there were lots of positives at each session.”


“…and their animation when they’re speaking is so hopeful… for

instance like a mum who would always sit on the settee and not

give very much, but when we did the solution-focused stuff she

ended up walking me to the door, which i know sounds small,

but she would never do that.”


Providing parents with the opportunity to reflect on the progress they had made from session to session enabled them to recognise the improvements they were capable of making – and were already making - in their lives. Practitioners were able to identify changes in a parent’s mood during particular sessions when they had helped them to focus on what they were already doing well. This was perceived as being particularly valuable in circumstances where parents were more accustomed to receiving negative feedback about their children and their performance as parents:

“i think she was very lacking in confidence as a parent, especially

when you’ve got a child who is on a part-time timetable at school

because of their behaviour; people telling you it’s your fault and

you need to do this and you need to do that. Actually looking at

her strengths made her feel better.”


Gaining an insight into the child’s perspective

The individual work with parents allowed them to understand how much of their behaviour their children were likely to notice and how that might make them feel. This could be the first time they had gained an insight into their children’s experiences and understanding of life at home. It was a turning point, helping parents to understand


and consequently communicate with their child, which in turn helped to strengthen their relationships:

“it kind of made me think about like how things are for them. so,

if i’m having a bad day then it’s kind of made me think about how

they might be feeling. they might be feeling scared or worried or

that they might have to kind of make up for that and stuff like that”


“Without it, we’d still be struggling and still be different, and i

wouldn’t have as close a relationship as i’ve got with her, and

that’s such an important thing. it helped with the understanding

for us both because i know how she feels now and she knows

how i feel, instead of us butting heads and just ignoring each

other.”


The structure of Family SMILES, involving work with both the parent and child, also played a key role in ensuring that parents heard the voices of their children. Parents and practitioners described how powerful it could be for parents to hear children’s feelings and experiences directly in their own words:

“We came together and went through it, and i read it out to them

and i think that was quite emotional because i was having to say

things that the child had said in the smiles group, for instance:

‘i’m sad when my mum smashes something precious of hers and

i have to pick it up.’ And.so i think that was difficult for the mum

to hear but it was really good for the child to be heard.”


While hearing their children describe life at home could be painful for parents, it was identified as an important facilitator for prompting change in parents’ understanding as well as their own behaviour.

Developing a support plan as a family

A key tool that parents developed together with their families during Family SMILES was a support plan, which set out who the child could contact in case of an emergency, and which drew on the wishes and preferences of the child. During interviews, these were described by both parents and children as something they continued to refer to and which provided reassurance to parents that the family would be able to cope should their mental health decline:


“We’ve now got rotas in place, we’ve got action plans in place,

they’ve even got little rotas in their rooms with things to do, so

when i’m not functioning fully at least the family can. it’s actually

taught her to cope with things and it’s taught me that they can

cope with things.”


“now we’ve got the action plan together, it gives me so much

more confidence because for instance, my partner and my mum

were both in hospital last week, so that’s quite daunting for me

because they’re my support network, so without them it’s quite

difficult. But the kids were able to step up and i know they’ve got

a support plan, if something goes wrong and i can’t cope they

can cope, they know who they can ring, they know they’ve got

their action plan and it works really, really well.”


The process of drawing up the support plan not only brought the whole family together to share their concerns and wishes, the final plan also provided parents with the reassurance that the family would manage in case of a crisis. This was a key facilitator of change for parents.

Flexible and reassuring approach of practitioners

Parents liked the flexible approach of NSPCC practitioners to the individual work. They particularly valued practitioners being responsive to the state of the parent’s mental health during each session and recognise when other issues needed to be addressed ahead of the planned content for that particular session:

“if she [practitioner] came and i was on a down day, sometimes

i wasn’t in the frame of mind to be able to deal with what she

wanted to talk about, but she’d give me the time to deal with

what that issue was and see how we could go around dealing

with that at that time.”


Apart from the flexible approach taken by practitioners, parents and carers also spoke about the reassuring and caring support they provided, and the importance of this in helping them to see the programme through and to feel positive about their participation in it:


“it’s been very, very traumatic for the children – it has for us. you

know, we’re grandparents but we’re not child psychologists,

and at times i used to say to them ‘i’m fearful of what to say in

case i say the wrong thing and make it worse.’ But when you’ve

got people you can turn to and who can give you that advice it

makes you… feel reassured.”

Carer, daughter (parent) is bipolar

“she was friendly and down to earth, she wasn’t sort of lording

it over anyone, or: ‘i’m from the nspcc and i’m here to...’ We

were just talking; talking like you would with a friend or somebody

else like that. she knew her stuff and it didn’t come out as she

was trying to tell you things, it was like she was trying to suggest

things to help and she wanted to help.”

Parent, bipolar

It was not only the practitioners’ approach to interacting with parents that was identified as important for facilitating positive change in parents’ lives, but also their role as advocates for the parents when dealing with other agencies:

“they’ve supported me by ringing the social worker and saying,

‘hold on a minute, this isn’t right, in our opinion…’ and they’ve not

just been the children’s voice, they’ve been my voice as well.”

Parent, bipolar

Getting help in navigating their relationships with other agencies was particularly important to parents, as they had previously felt they were not being listened to or represented fairly.

3.5.1 Barriers

Parent not able to acknowledge their mental health problems

In circumstances where parents were not able to recognise they had a mental health problem and that this could be affecting their parenting, it was often difficult for practitioners to engage them successfully in the work:

“she says to me: ‘i’m not depressed, i just don’t always cope.

my problems are with the eldest daughter and she causes the

problems in this house and i’d manage if she didn’t live here; i

would be able to cope.’”



Some parents felt that the problems they faced as a family were not related to their mental health, but rather due to other difficulties. Furthermore, where the mention of mental health caused offence to parents it was a challenge for practitioners to convey the importance of the programme to them:

“there was one parent that i worked with who refused to accept

that they had a mental illness. When i spoke to her about it she

said that i was being rude telling her she was ‘mental’, you know,

like crazy.”


Parents who were unable to recognise that they had a mental health problem – let alone that this might be affecting their ability to meet the needs of their children – presented a significant barrier to carrying out the work and to enabling outcomes for parents to be realised.

Underlying problems in the parent-child relationship

In families where the parent and child relationship was very complex and problematic, focusing on the impact of parents’ mental health did not always make a difference to them or their children. One parent described the relationship with their child as ‘irretrievably broken down’, and in such circumstances, it was not necessarily possible for Family SMILES to address the issues that had contributed to the relationship difficulties:

“When you’ve got a difficult child and a parent who really doesn’t

put any boundaries in and stuff like that, it does leave it not even

half done.”


Similarly, in some cases parents’ negative attitudes towards their children were so entrenched that, without spending time exploring those issues, it was not always possible for the practitioners to bring about any improvements to communication or the parent-child relationship. In some circumstances, parents perceived their children as the source of the problems that they experienced and tackling such perceptions was particularly challenging for practitioners. In other families, parents described how their children did not respect them or expressed their dislike for them and these situations made it difficult for the Family SMILES programme to have a significant effect:


“i think it probably helped other families probably better than it has

here because of the lack of respect that goes on in this house.”

Parent, personality disorder

“even now i still love them because they’re my children, and if

i’m not there for them then who’s going to be there for them,

because their dad’s not there, but they love their dad more

than me. yes, it’s sad really, because this elder one says to me

‘you’re a biological mum, but not a real mum.’”


Where the relationship between the parent and child was very complex at the outset of the work, practitioners did not feel that it was always possible to support a process of positive change.

Ongoing and complex family circumstances

Parents’ mental health problems were often linked to complex life experiences they were struggling to cope with – for example, being in an abusive relationship; having a family member in prison, or seeking asylum in the UK:

“mum was too busy kind of fighting fires to then really be thinking

about doing smiles.”


In circumstances where there was no end in sight to the ongoing difficulties parents were facing, the parenting work was less likely to be effective. Where families had fled danger in their home countries and were caught up in lengthy asylum applications, practitioners suggested that they were not yet able to benefit from a programme such as Family SMILES, as their more immediate needs of safety and security were not being met. Similarly, in circumstances where families were living in inappropriate accommodation at the time of their engagement in their programme, this also presented a barrier:

“they get a glimmer of hope and then it’s taken away, and then

mum’s mental health deteriorates and it’s just a vicious circle and

there’s nothing – the only thing that can bring them out of that is

for them to be granted status [refugee status], which is a lengthy

process. you know, they have been in this country i think for four

or five years now and it’s still ongoing.”

Referrer, Voluntary Sector Organisation


“they were living in a terrible house, lots of crime and drugs and

racism that they were dealing with every day. it was frightening

and they were all living in the same bed because they were too

worried about their environment. so yeah, i can see it would have

been very hard to focus on us.”


Where parents continued to feel anxious and preoccupied by ongoing and immediate difficulties, Family SMILES could not be made their primary focus, and the content of the work may have felt less relevant and helpful in such circumstances.

Concern about stigma around accessing NSPCC support

A further barrier for some parents was the knowledge that Family SMILES was being delivered to them by the NSPCC, and their own preconceptions of what this meant for them and their children:

“initially i was petrified of [the] nspcc. i mean, i didn’t know

smiles was with the nspcc at first. i probably would have run

a mile, because initially you think that’s children that are being

abused. it is a big stigma.”


During interviews, some parents described how they had intentionally hidden from their partners the fact that they were receiving help from the NSPCC, or received negative reactions from friends and family about accessing NSPCC support. It was suggested that the stigma attached to the NSPCC might discourage some parents from accepting referrals to Family SMILES:

“there’s loads of other families that could do with this. i know of

loads myself and they just would not dare, there is no way they

would ask for them (nspcc) to be involved, because you look

at the things on the telly and all it is, is coming and taking your

kids away”


However, this stigma of receiving NSPCC support presented more of an initial barrier: once parents had met with practitioners and understood that the programme aimed to help strengthen their family, they were more enthusiastic and willing to engage.


In summary

The key findings relating to facilitators and barriers to positive change for parents are:

• Facilitators for change included:

– opportunity to reflect on the impact of difficult life events

– focusing on parents’ strengths

– gaining an insight into the child’s perspective

– developing support plans as a family

– flexible and reassuring approach of practitioners.

• Barriers to change included:

– parents not being able to acknowledge mental health problem

– underlying problems in the parent-child relationship

– complex, ongoing adverse family circumstances

– concern about the stigma around receiving NSPCC support.


Chapter 4: Exploring sustained change

4.1 Sustained changes for children and young people

Children’s emotional and behavioural difficulties at six-month follow up: SDQ

Sixty-five SDQs were completed by children at all three time points: Time 1 (start of the intervention); Time 2 (end of intervention); and Time 3 (six months after the intervention was completed).

The mean SDQ score decreased from 16.23 at Time 1 to 15.42 at Time 2, and further down to 15:03.at Time 3, a decrease that might suggest that children’s emotional and behavioural difficulties reduced over the course of Family SMILES and beyond. However, as the decrease was not statistically significant, it does not indicate a general improvement in children’s emotional and behavioural difficulties between the start of the programme and six months after completing it.

Understanding change across clinical and non-clinical bands

Clinical change has also been explored for this cohort of children. The distribution of the data at Time 1, Time 2 and Time 3 is given in Table 11.

Table 11: Proportion of children with clinical and non-clinical SDQ scores from Time 1 to Time 3 (n=65)

T1 T2 T3

Non-clinical 54% 68% 71%

Clinical 46% 32% 29%

The table above highlights a proportional shift into the non-clinical band by the end of the programme and again six months after programme completion. The 54 per cent of children who were in the non-clinical band at the start (Time 1) increased to 68 per cent by the end of the programme (Time 2), and to 71 per cent six months later (Time 3). This suggests that the Family SMILES programme resulted in children’s clinical improvement that was sustained beyond the end of the programme.

In Table 12 below the journey of children into the clinical or non-clinical bands at Time 3, is followed through from their starting points at Time 1 and Time 2.


Table 12: Children’s post-programme progress (SDQ)

Banding at T1 and T2 (N = 65) Non-clinical T3 (n=46)

Clinical T3 (n=19)

Non-clinical T1 and T2 (N = 31) 30 1

Non-clinical T1, Clinical T2 (N = 4 ) 2 2

Clinical T1 and T2 (N = 17) 5 12

Clinical T1, Non-clinical T2 (N = 13) 9 4

The data indicates that:

• Almost half of children in the sample (48 per cent) started and ended Family SMILES within the non-clinical range of emotional and behavioural difficulties. Only one child from this cohort had deteriorated, crossing the threshold into the clinical range at Time 3.

• Only a small number of children who started with a non-clinical level of need at Time 1 deteriorated into the clinical threshold by Time 2 (n=4). At Time 3, six months after finishing the programme, two of the four children had improved to a non-clinical level of concern.

• Of the 17 children who started and ended Family SMILES within the clinical range, five had moved into the non-clinical range at Time 3, suggesting the possibility of improvement for children beyond the end of the programme.

• Thirteen children improved from a clinical level of need at Time 1 to a non-clinical level at the end of the programme (Time 2). The majority (nine) were still within the non-clinical range six months after completing the programme (Time 3), showing that for these children, the clinical improvement brought about by Family SMILES was sustained beyond the end of the programme. However, as the sample of 13 children accounts for only 20 per cent of the total, it is not possible to undertake more robust analyses to examine sustained clinical change.

Children’s self-esteem: Rosenberg Self-Esteem questionnaires (Adapted)

Sixty-nine adapted Rosenberg Self-esteem questionnaires were completed pre- and post- intervention and again at six-month follow up. Figure 6 shows the average scores at each time point.


Figure 6: rosenberg Self-esteem questionnaire (Adapted) – mean scores pre- and post-Family SMILES and at 6-month follow up

22

20

18

16

14

12

10

8

6

4

2

0 T1 T2 T3

The Rosenberg mean score increased from 18.4 at Time 1 to 19.5 at Time 2 and to 19.9 at Time 3, which would seem to suggest that children’s self-esteem generally increased through Family SMILES, and that this improvement was sustained beyond the end of the programme. However, as the difference between the mean score at Time 2 and Time 3 was not statistically significant, we cannot confidently say that this is so.

4.2 Sustained changes for parentsCAPIs were collected from 22 parents at all three time points. Of the 22 sets, 18 were valid at all three time points (82 per cent).

Sustained change in behaviours that contribute to risk: CAPI

Due to the small sample, it was not possible to run any statistical analysis on this data, so descriptive analysis has been used instead to explore changes for this cohort of parents. The table below sets out the mean scores for each CAPI subscale at Time 1, Time 2 and Time 3.


Table 13: Average scores reported by parents who completed the CAPI pre- and post- programme and at 6-month follow up (n=18)

Pre-programme (T1)

Post-programme (T2)

Six-month follow up (T3)

Total score (cut-off point of 215) 319.56 304.39 288.17

Distress score (cut-off point of 152) 209.72 204.72 193.89

Rigidity score (cut-off point of 30) 17.72 17.67 14.83

Unhappiness score (cut-off point of 23) 42.94 33.61 31.72

Problems with family (cut-off point of 18) 20.89 20.28 20.67

Problems with child (cut-off point of 11) 10.33 9.5 8.78

Problems with others (cut-off point of 20) 17.94 17.94 18.28

Ego strength 6.22 7.67 9.67

Loneliness scores 12.61 12.78 11.78

Due to the very small sample size for this cohort of parents, the following findings should be interpreted with caution:

• Total scores decreased by the end of the programme and had decreased again at the point of six-month follow up. This suggests that the improvements experienced by parents continued after they completed Family SMILES.

• However, the mean total score remained above the clinical threshold, suggesting that while the improvements were sustained, the concerns remained at a clinical level.

• Similarly, there was an improvement in mean scores, continuing to improve beyond the end of the programme, in levels of distress and unhappiness, which however remained above the clinical cut-off points at Time 3. The same applies when comparing the pre- and post-programme data sets for these two subscales: while the improvement was statistically significant, the problems remained within the clinical range.

• The scores for ‘rigidity’ and ‘problems with child’ continued to improve at Time 3, suggesting sustained improvement in these areas. However, these scores were already below the clinical threshold at Time 1, indicating that they may not have been areas of great concern for parents from the outset of Family SMILES.

• ‘Problems with family’ had improved slightly at Time 2, but then deteriorated again, albeit only a little, over the following six months. At all three time points, scores remained above the clinical threshold, suggesting continuing difficulties for parents in their relationships with other family members.


• Similarly, scores for ‘problems with others’ had remained the same at Time 2 but then deteriorated at Time 3. However, these scores remained below the clinical threshold, suggesting that this was not a problematic area for parents in the first place.

• ‘Ego strength’ continued to improve beyond the end of the programme, indicating growing confidence among this cohort of parents.

• While levels of loneliness had increased by the end of the programme (Time 2), it seems that another six months on (at Time 3) parents felt less lonely than they did to begin with (Time 1).

In summary

The key findings from this chapter on sustained change are:

• While changes in SDQ scores from the start of the programme to completion and then at six-month follow up do appear to indicate a sustained improvement in children’s emotional wellbeing and behaviour, the changes are not statistically significant, so we cannot be confident that the programme consistently produces these positive outcomes.

• Mean scores indicate an improvement in children’s self-esteem at the point of six- month follow up. However, again, this change is not statistically significant.

• There appears to be a sustained change in parents’ protective parenting behaviours, but this remains an area of difficulty within the clinical range for the CAPI. The sample size for parents at Time 3 was very small, so no statistical testing on this data set was carried out and this analysis should therefore be interpreted with some caution.


Chapter 5: Conclusion and discussionThe evaluation of Family SMILES has highlighted positive change for both children and parents at the end of the programme. There was a small but statistically significant improvement in the anticipated outcomes for children; reduced emotional and behavioural difficulties, improved self-esteem and an improved ability to process thoughts and feelings. Changes in children’s emotional and behavioural difficulties were also clinically significant. A clinically significant change was defined as one that moved from the clinical to non-clinical range on the SDQ. Changes for parents against the outcomes articulated for them in the theory of change - improved protective parenting and greater insight into impact of mental health problems on children – also saw a statistically significant improvement. However, parents’ CAPI scores remained within the clinical range at the end of Family SMILES, suggesting that these changes did not move parents below the threshold of clinical concern. This finding is discussed in further detail later in this chapter.

The findings are supported by comparison with a small group of families who had been assessed for Family SMILES, attending up to four assessment sessions with an NSPCC practitioner, but who were waiting to be allocated to an appropriate Family SMILES group. This comparison highlighted a greater level of change, including clinical change, for parents and children who received the service compared to those who had only completed the assessment. This suggests that the outcomes for parents and children identified at the end of the programme can be attributed to Family SMILES since the same extent of improvement was not apparent for parents and children who had not received the service.

Follow-up data at Time 3, six months after programme completion, presents a mixed picture of sustained change for parents and children. While children’s data seemed to indicate a pattern of scores improving at each time point, these changes were not statistically significant. Time 3 data for parents indicated continued improvement in mean scores beyond the end of the programme, but remaining at a clinical level; it was not possible to carry out statistical testing on this data since the sample size was too small.

Despite the limitations noted above, this study has found potentially significant benefits for families taking part in Family SMILES, in particular: improving understanding and communication about mental health within families; increased parental awareness of the impact of their mental ill health on their children; and children’s increased


confidence and self-esteem. The significance stems in part from the potential for initiating a process and a change in family dynamics that builds greater openness and honesty, which should help families to navigate challenges, including those arising from external stressors.

5.1 Suggestions for improvementThere were some suggestions from participants and professionals on how the programme could be improved. These included:

• A review of the assessment process is needed to ensure that children are appropriately matched to groups.

• All children taking part in the programme should have a clear understanding that their parent has a mental health problem.

• Families and professionals felt it was important to ensure that the locations of group sessions are accessible.

• Transport should be provided to those families who might struggle to attend sessions without such support.

• The content of group sessions should include, for example, discussions on domestic violence.

• It would be a good idea to have mental health professionals present at groups to broaden children’s understanding of mental health and the way various problems can be treated.

• Families felt that the service should be made available to a broader age range of children.

• Parents stressed the importance of scheduling the group sessions to avoid clashes with other similar services in the local area, such as Young Carers, a group that some children who attended Family SMILES had also been involved with.

• Parents in particular suggested promoting Family SMILES more widely, within an NHS setting. They felt that improved publicity of Family SMILES would enable more families to access valuable support.

• Both children and parents mentioned that they would have valued the provision of longer term or drop-in support after completing the programme, or signposting to other services so that they might access such support on a more ongoing or ad hoc basis.


5.2 Discussion

5.2.1 Parents’ level of clinical need at the end of Family SMILES

CAPI data has highlighted that while there was significant improvement in parents’ protective parenting by the end of the programme, the mean scores remained at a clinically problematic level. This change was also reflected in follow-up data for the CAPI, with mean scores decreasing but remaining within the clinical range. The fact that parents’ levels of difficulty remained at a clinical level at the end of Family SMILES as well as at six-month follow up points to limitations on the extent of improvement brought about by Family SMILES for parents.

It is important to take into consideration that two key domains of the CAPI remained at a particularly high level at the end of the programme, even though significant improvement had taken place in these areas: parents’ unhappiness and distress. This is particularly pertinent, as parents with ongoing mental health problems might be expected to display a clinical level of need in these areas, so this should be taken into account when interpreting the data. Moreover, the fact that there is an overall significant improvement for parents does indicate a greater capacity among parents to protect their children from harm by the end of the programme. Family SMILES is, after all, not a therapeutic service for parents, so the programme should not be expected to reduce symptoms associated with parents’ mental health needs. This, then, may provide some explanation as to why these domains remained at such a high level, affecting the overall CAPI scores for parents at the end of the programme.

It has not been possible to compare this data with other CAPI data collected from populations of parents with significant mental health problems. Such a comparison might provide further insight into the normality of the data set and the average scores in these domains for such groups of parents.

5.2.2 The role of Family SMILES in the context of ongoing external stressors

The qualitative element of this evaluation has highlighted a barrier to outcomes being achieved by families as they struggle with ongoing stress factors, which have a profound impact on parents’ mental health. Examples of such external causes of stress that emerged from interviews included dealing with the trauma of fleeing danger in families’ home countries and facing the asylum process in the UK; poor accommodation; domestic violence, and ongoing substance


misuse. Where families were faced with such ongoing difficulties it proved very difficult for them to engage effectively in the programme or to feel that the work was meeting their needs, since these were so complex and entangled in the daily difficulties they faced.

While it is apparent that these environmental factors do affect parents’ capacity to engage in the programme, it is less clear what the effects are on their children and the extent to which they are able to benefit from Family SMILES. This is an area that may require further investigation in order to determine any implications on the inclusion and exclusion criteria for the programme.

5.2.3 Peer problems and the peer experience as a facilitator of change

A key facilitator of change identified for children was the opportunity to meet other children whose parents also had mental health problems and who were experiencing similar difficulties to themselves. This finding around the importance of the peer experience is particularly pertinent within the context of over half of children continuing to report a clinical level of need in the peer-problems subscale of the SDQ at the end of the programme.

On the one hand, this suggests that while Family SMILES may have provided a positive peer experience for children, they may still have been experiencing ongoing difficulties with friends and peers at school and in other environments external to Family SMILES. It is also possible that children’s feelings about their positive peer experience on the programme conflated with their emotions around the group work ending at Time 2, causing them to reflect negatively on the peer problems they faced, knowing that they would no longer be part of the Family SMILES group. This links to findings around some children struggling to accept the group coming to an end, and the extent to which this acts as a barrier to outcomes being achieved for children at the end of the programme. It highlights the potential for more ongoing support to be made available for children, particularly within a peer environment, once Family SMILES has ended.

5.2.4 Improving communication within the family

One outcome that the evaluation did not specifically measure but that was reflected in the qualitative interviews related to an improvement in family communication post-Family SMILES. While the eligibility criteria for the programme require the child to be aware of the parents’ mental health problems, often these had not been acknowledged or discussed with the child within the family before starting Family SMILES.


Families described how the programme had given them the language to talk about mental health more appropriately with each other, arising from a greater knowledge about terminology (medication, treatments and conditions) and their meanings. In addition to language, children alluded to having a better understanding of their parent as helpful in enabling them to respond confidently and appropriately to their parent, with more care and respect.

Being able to talk more openly and easily about mental health provided families with a sense of relief that there were no longer secrets within the home environment. While prior to engagement in Family SMILES, discussions about mental health may have caused embarrassment, the programme enabled parents and children to understand one another better, and provided them with the skills and insight to respond to each other with more empathy, and to talk comfortably about feelings as a family, strengthening the relationship between the child and the parent. This is potentially very significant, as it may have acted as a catalyst for further improved communication within the family, with parents more able to continue discussions with their children about their mental health problems, and children communicating their needs and concerns more confidently within the family environment.

5.2.5 Bridging the gap between adult mental health and children’s services

The evaluation highlights the value of an approach to service delivery that works with the whole family, bringing together adults and children who may previously have been engaged with multiple agencies. This suggests that the Family SMILES programme can bridge a gap between adult mental health and children’s services by creating a safe space for parents and children to explore the impact of mental health on the family.

This linking of adult mental health and child protection support draws attention to the value of a service such as Family SMILES for all families where there are parental mental health problems, regardless of whether there have been any prior child welfare concerns. Providing parents with support to understand the potential implications of their mental health problems on their children’s wellbeing, and enabling children to develop their understanding of mental health and how to keep safe at home, presents a valuable model for improving communication and relationships within the family environment.


5.2.6 Making change sustainable: reviewing the theory of change and future developments

While the Family SMILES programme has some strengths, the evidence from this study suggests that the underlying theory of change needs to be revisited. A key question relating to children, for example, concerns the reduction of risk. The fact that the improvement in parents’ protective behaviours was not clinically significant suggests that the aim of risk reduction had not been sufficiently met. For parents, although they felt happier and less distressed at the end of the programme, difficulties remained at a clinical level. This is potentially due to the ongoing mental health problems they faced, which in turn may be underpinned by broader societal and structural factors such as poverty and poor housing. It is unlikely that the Family SMILES service would, in isolation, be able to address these broader issues sufficiently in order to show a clinical improvement among service users.

A review of the programme’s underlying theory of change should also consider its position within the wider service network. While it is encouraging to see, for example, that parents felt more able to ask for support, this will only lead to further change if that support is actually available. This is likely to have a significant impact on the degree to which any outcomes are sustained.

To summarise, the revised theory of change should address:

• how a reduction in risk for children will be achieved

• the broader structural and societal factors that can impact on parents’ mental health and the family’s experience of it

• the wider systems and networks within which Family SMILES operates, which will be critical for achieving and sustaining positive outcomes.

The NSPCC is currently testing this model of programme delivery in partnership with Manchester University. Family SMILES has been adapted into a programme called Young SMILES and this is being delivered by the NSPCC and within an NHS context. The eligibility criteria have been changed to work with a wider age range of children, who may or may not have been identified as being the subject of child protection concerns, but who have a parent with a serious mental health problem. A feasibility study for a randomised control trial of the adapted service is currently underway, and will build on the evidence that has been generated through this evaluation of Family SMILES.


The need to address the structural, societal and system factors highlighted above will also be addressed through a new initiative the NSPCC is promoting in selected areas: Together for Childhood. There will be three projects aimed at families experiencing multiple adversities, including mental health difficulties, drug and alcohol abuse and domestic abuse. A public health approach will be adopted, promoting prevention at different levels. The ethos of the projects will be collaborative, to work in partnership with local communities and agencies. It is hoped that these projects will be transformative in their local areas by generating systems and cultural change through workforce development, campaigns and empowering local communities and service users to set priorities and oversee progress.


BibliographyAldridge, J. (2006) The experiences of children living with and caring

for parents with mental illness. Child Abuse Review 15:79-88.

Aldridge, J. (2011) Children living with parental mental illness: briefing. Stirling: Scottish Child Care and Protection Network.

Baldwin, P. and Glogovic, C. (2010) Providing S.M.I.L.E.S to children of the Waterloo region whose caregiver has a mental health diagnosis. Ontario Association of Children’s Aid Societies Journal, 55(1).

Bee, P., Bower, P., Byford, S., Churchill, R., Calam, R., Stallard, P., Pryjmachuk, S., Berzins, K., Cary, M., Wan, M., Abel, K. (2014) The clinical effectiveness, cost-effectiveness and acceptability of community-based interventions aimed at improving or maintaining quality of life in children of parents with serious mental illness: a systematic review. Health Technology Assessment 18 (8). Available from: www.journalslibrary.nihr.ac.uk/__data/assets/pdf_file/0019/106453/FullReport-hta18080.pdf [accessed 10 May 2014]

Brandon, M., Sidebotham, P., Bailey, S. and Belderson, P. (2011) A study of recommendations arising from serious case reviews 2009-2010. London: Department for Education (DfE).

Crossroads Caring for Carers and The Princess Royal Trust for Carers (2008) Economic Evaluation of Young Carers Interventions. Available from: http://static.carers.org/files/finalfinal3-4040.pdf [accessed December 2015]

Darlington, Y., Feeney, J. A. and Rixon, K. (2005) Interagency collaboration between child protection and mental health services: practices, attitudes and barriers. Child Abuse and Neglect 29:1085–1098.

Dearden, C. and Becker, S. (2000) Growing up Caring: Vulnerability and Transition to Adulthood – Young carers’ experiences. Youth Work Press, for the Joseph Rowntree Foundation.

Dearden, C. and Becker, S. (1995) Young Carers: The facts. Sutton: Reed Business.

Department of Health (DH), Department for Education and Employment (DfEE) and Home Office (2000) Framework for the assessment of children in need and their families. London: The Stationery Office (TSO).

http://www.oacas.org/pubs/oacas/journal/2010SpringSummer/smiles.html



http://www.journalslibrary.nihr.ac.uk/__data/assets/pdf_file/0019/106453/FullReport-hta18080.pdf

http://www.journalslibrary.nihr.ac.uk/__data/assets/pdf_file/0019/106453/FullReport-hta18080.pdf


Eastman, B. and Bunch, S. (2009) Variables associated with treatment outcome in a sample of female offenders. The Open Criminal Journal 2:18-23

Fudge, E., and Mason, P. (2004). Consulting with young people about service guidelines relating to parental mental illness. Australian e-Journal for the Advancement of Mental Health, 3(2).

Gelkopf. M. and Jubotaro, S. E. (2013) Parenting Style, competence, social network and attachment in mothers with mental illness. Child and Family Social Work Volume 18, Issue 4 pp.496 – 503.

Goodman, R. (1997) The strengths and difficulties questionnaire: a research note.. (Journal of child psychology and psychiatry and allied disciplines, Vol.38, Iss.5) pp 581-586

Gorin, S. (2004) Understanding what children say: children’s experiences of domestic violence, parental substance misuse or parental health problems. London: National Children’s Bureau for the Joseph Rowntree Foundation.

Henshaw, C. (2011) Patients as Parents: supporting the needs of patients who are parents and their children. London: Royal College of Psychiatrists.

Maybery, D., Reupert, A., and Goodyear, M. (2006). Evaluation of a model of best practice for families who have a parent with a mental illness, available from: https://www.easternhealth.org.au/images/services/champsreport.pdf

Milner, J.S. (1986) The child abuse potential inventory manual. 2nd edn. Dekalb, Illinois: Psytec.

Morris, R. (1981) Society and the Adolescent Self-Image. Revised edition. Middleton, CT: Wesleyan University Press.

NSPCC (2015) Parents with a mental health problem: learning from case reviews. London: NSPCC. Available online: https://www.nspcc.org.uk/preventing-abuse/child-protection-system/case-reviews/learning/parents-mental-health-problem/[Accessed 28/06/17]

Office of National Statistics. (2001) Psychiatric Morbidity Survey. London: ONS.

Pitman, E. and Matthey, S. (2002) Evaluation of the SMILES program for children with mentally ill parents: Bankstown, Sydney. South West Sydney Area Health Service, Sydney, Australia.

https://www.easternhealth.org.au/images/services/champsreport.pdf

https://www.easternhealth.org.au/images/services/champsreport.pdf

https://www.nspcc.org.uk/preventing-abuse/child-protection-system/case-reviews/learning/parents-mental-health-problem/




Pitman, E. (2010) SMILES program: simplifying mental illness plus life enhancement skills Adelaide S. Aust.: Children Of Parents with a Mental Illness (COPMI).

Prokop, J., Bundred, K. and Green, J. (2010) Background paper on London serious case reviews completed April 2006 - September 2009. London: London Safeguarding Children Board.

Ritchie, J. and Lewis, J. (2003) Qualitative Research Practice: A guide for social science students and researchers. Sage Publications: London; Thousand Oaks; New Delhi.

Roscoe, H., Constant, H. and Ewart-Boyle, S. (2012) Think child, think parent, think family: final evaluation report (Family and children’s services report, 56) London: Social Care Institute for Excellence (SCIE).

Roy, R. (1990) Consequences of Parental Illness on Children: A Review. Social Work and Social Sciences Review 2(2)

Royal College of Psychiatrists (1996) Health of the Nation Outcome Scales (HoNOS). London: Royal College of Psychiatrists

Sahin, N. (2010) Childhood Trauma, Types of Marriage and Self-Esteem as Correlates of Domestic Violence in Married Women in Turkey. Journal of Family Violence 25 (7):661-668.

Spade, Julie A. (2007)The Relationship Between Student Bullying Behaviors and Self-Esteem. Electronic Thesis or Dissertation. Bowling Green State University. Available from: https://etd.ohiolink.edu/

Stallard, P., Norman, P., Huline-Dickens, S., Salter, E. and Cribb, J. (2004) The effects of parental mental illness upon children: A descriptive study of the views of parents and children. Clinical Child Psychology and Psychiatry 9:39-52.

Tunnard, J. (2004) Parental Mental health Problems: Key Messages from Research, Policy and Practice. Dartington: Research in Practice.

Wan, M.W., Warburton, A.L., Appleby, L. and Abel, K.M. (2007) Mother and baby unit admissions: feasibility study examining child outcomes 4–6 years on. Aust NZ J Psychiatry 41:150–6.

http://www.copmi.net.au/jsp/resources/files/smiles_program/app/program_info_p97-101.pdf

http://www.copmi.net.au/jsp/resources/files/smiles_program/app/program_info_p97-101.pdf

http://www.londonscb.gov.uk/files/2010/resources/serious_case_reviews/scrs_in_london_for_web.doc

http://www.londonscb.gov.uk/files/2010/resources/serious_case_reviews/scrs_in_london_for_web.doc

https://etd.ohiolink.edu/


Appendices

Appendix 1: Programme details

Overview of sessions

Overview of sessions for children:

SESSION 1: All about me

AimsThe child feels settled in the group and gains an understanding of the programme.

Learning outcomesThe child identifies strengths and resilience factors in their life, such as people they like and things they enjoy.

SESSION 2: Mental illness

AimsThe child begins to understand mental illness, and works with the group to dispel some myths.

Learning outcomesThe child has a better understanding of mental illness and the terminology around mental illness.

SESSION 3: My Story

AimsThe child begins to understand how mental illness affects their family.

Learning outcomesThe child sees how their parent’s mental illness leads to them assuming a particular role within the family. The child identifies positive and negative aspects of this role, relating to their own health, wellbeing and development.

SESSION 4: My feelings

AimsThe child explores and understands the emotional impact mental illness has on them and their family.

Learning outcomesThe child identifies strengths and resilience factors in their life, such as people they like and things they enjoy.


SESSION 5: Looking after myself

AimsThe child increases their ability to maintain and improve self-esteem.

Learning outcomesThe child learns, and can employ, self-esteem and assertion techniques, to help if they are feeling self-critical or have been criticised or victimised by others.

SESSION 6: People I can trust

AimsThe child identifies people who can support them if they are feeling upset, lonely, sad or confused.

Learning outcomesThe child reflects on the people who can provide support when needed and thinks about how those people can offer support. The child collates information that will contribute to the support plan component of Family SMILES.

SESSION 7: Identifying sources of support/safety planning

AimsTo assist participants in identifying support, and in thinking through what sources of support they might use in different situations.

Learning outcomesChildren understand that it is the responsibility of adults to ensure their safety and are able to identify the people within their own support system whom they can rely on if needed.

SESSION 8: Feedback, evaluation and graduation

AimsTo reflect on feelings about group ending and formally evaluate the programme.

Learning outcomesThe graduation enables children to take pride in completing the group and to bring together any reflections on being part of the group.


Overview of parent sessions:

SESSION 1: My story

Aim of sessionTo help the parent gain an insight into their illness, how it affects them, their relationship with their child and other significant family members.

SESSION 2: Managing anxiety

Aim of sessionTo help the parent gain an understanding of how anxiety and stress impact upon them and their child.

SESSION 3: My parenting

Aim of sessionTo assist the parent in setting boundaries, examining what helps and what hinders boundary setting.

SESSION 4: My strengths as a parent

Aim of sessionTo discuss and reinforce what the parent does well.

SESSION 5: Understanding my child’s feelings

Aim of sessionTo increase the parent’s understanding of how their child experiences family life.

SESSION 6: Communicating with my child

Aim of sessionTo increase the parent’s understanding of how to effectively communicate with their child.

OutcomeThe parent will have knowledge of practical strategies to aid effective communication with children.


Programme inclusion and exclusion criteria

Inclusion:

• The child has a parent (or significant carer) with a mental illness, which has given rise to, or contributed to, concerns about their ability to safely care for the child.

• The parent has been diagnosed or is being assessed for diagnosis by health professionals as having a mental illness or has been receiving professional support for a significant period for a mental illness, but has not received a diagnosis.

• The child is aware of the parent’s illness.

• The parent is informed of the purpose and remit of the group and consents to their child’s attendance.

• The child falls within the age limit for the group and has been assessed as cognitively able to participate in the group.

• The child presents as being emotionally able to engage in the group. Children with emotional/behavioural difficulties who are receiving a service from CAMHS can be considered for participation in the programme. However, advice should be sought from the relevant CAMHS team as to the child’s suitability for and ability to engage with the programme and it should be clarified that CAMHS remains the lead service with responsibility for the mental health of the child.

Grounds for exclusion:

• The child is not in a reasonably secure family/living position – i.e. either in the process of being admitted into care, or very recently admitted into care.

• The child presents with emotional or behavioural distress on a level that requires assessment by a suitably qualified healthcare professional. NSPCC assessment should be suspended, pending outcome and recommendations regarding the child’s ability to engage in the programme.

• The child or young person has a diagnosis of a major mental illness that significantly impedes their ability to participate safely in a group.

• If the parent is extremely unwell at the time of the group, attendance may not be appropriate due to corresponding distress levels in the child and inability of the parent to participate in individual work. This should be assessed on a case-by-case basis.


Appendix 2: Interview schedules

Interview schedule for children1. Genogram: To understand more about you and your family.

2. About the programme: To understand how much children remember about the programme and activities .

3. Specifically about the group experience

– How did you feel about working with the other children in your group?

Did you enjoy this most of the time or not? (explore: group dynamics)

(If yes) What did you like about this?

(If no) What didn’t you like about this?

– How did you feel about talking about things that have happened to you and your parent?

(If yes) Did you find it any easier to talk about things after you had been at the groups a bit longer?

Did you feel any better or worse after talking about these things?

4. Changes before and after:

4.1 How do you feel life is going for you overall?

- Did they make you feel differently about yourself?

4.2 Change in understanding about your parent’s mental health

- Do you feel you learnt anything about your parents’ situation?

4.3 Change in talking about your parent’s behaviour with others

- Did they make you feel differently about your parent?

4.4 Change in your own behaviour

- Have you noticed any changes in yourself at home, with friends or in school?

4.5 Change in your parents’ behaviour

- Do you think they feel better after this work or not?

4.6 Change in your relationships

- Have you noticed any changes in how you get on with others and your parents?


5. Comments on how the support plan worked

How did you feel about doing the joint work with your parent?

Did you enjoy this most of the time or not?

(If yes) What did you like about this?

(If no) What didn’t you like about it?

Is there anything that could be done to make the Family SMILES programme better? If yes, please explain.

Interview schedule for parents1) On the whole, how would you describe your experience

on this programme?

For example, has it been a mostly positive or negative experience?

2) Was there anything you particularly liked or disliked about the individual work?

Prompts used if necessary: For example, any particular sessions, anything in general about the sessions?

3) Do you think this experience has helped you in any way, or not?

Prompts used if necessary: For example, coping with your past experiences, relating to your child, understanding your child better, feeling better about yourself?

3a) (If yes) Please give details.

3b) (If no) Are there any reasons you feel this work has not helped you?

4) Do you think the groups/work have helped your child in any way, or not?

Prompts used if necessary: For example, coping with their past experiences, affecting their behaviour in any way, dealing with their emotions differently?

4a) (If yes) Please give details

4b) (If no) Are there any reasons you think they have not helped?

5) Do you feel that your relationship with your child has been affected in any way by the Family SMILES program?

5a) If yes, how has it been affected?

For example: positively or negatively?

6) How have you felt about talking about things that may have happened in your past to the worker?

For example: have you found it hard, easy. What has been hard/easy?


7) How have you felt about talking about your experiences with your child in the joint work?

For example: have you found it hard, easy. What has been hard/easy?

8) Has this programme made you think differently about anything that has been discussed?

8a) (If yes) What do you think you have learnt?

9) Do you feel you would know where to go for further support if you needed it?

Where do you think you would go to get this support?

10) What did you think of the worker on the programme?

For example: in terms of how well you related to them, how easy it was to talk to them, how they dealt with the group, how they dealt with sensitive issues?

11) Do you have any suggestions of how this service could be improved?


Interview schedule for Family SMILES practitionersTheme Need to explore

1. Introduction and sharing of specific cases (before the interview practitioners will be asked to think of a case where they perceived a high level of change for the family and one where there was no change for the family. This will be used as the starting point for discussion).

• Changes that took place for the case where a positive outcome was perceived: – What did these changes look like and when did they take place? Was there change for everyone in the family or just the parent? Child? Those family members receiving the Family SMILES service?

– Can the success of this case be attributed to any specific aspects of the programme?

– Can the success of this case be attributed to any aspects of the family’s circumstances at the point of starting the programme (i.e. improvement in parents’ mental health in recent months)

• Do you still have contact with the family? – Have these changes been sustained?

• The case where no changes/negative changes occurred: – Who took part in the programme? Any significant issues at home? – What do you think were the barriers to change coming about for the child/parent?

– Do you think any changes to the programme structure/content could have changed outcomes for that family at all?

• Do you still have contact with the family? Do you know if there have been any changes in their situation since completing Family SMILES?

• How these cases compare to other children/parents you have seen come through the programme.

2. Programme structure: Exploring the experience and value of each component of the programme – group work, parent work and joint work.

• Experience of delivering each component of the programme? i.e. parent work, group work, joint work.

• Views on the strengths and challenges of each component of the programme?

• Any particular group work sessions/activities that were helpful in bringing about outcomes for children? Any sessions that stand out as being less helpful?

• Any particular parent work sessions that were especially helpful for improving protective parenting? Any that were less helpful?

• Usefulness of manual for carrying out sessions with parents and children?

• Anything that would have helped to improve delivery of this programme?

• Views on key skills needed to be able to deliver Family SMILES effectively?

3. Implementation issues

• Process of work – creating groups of children with similar ages, managing demand over geographical area, managing endings, referring on.

• Support structures needed to deliver the service.

4. Access to the service

• What helped referrals come in or not? • What were the views of referrers?

5. Drop-out from programme and long term changes

• Most common reasons for children/parents to drop out from the programme

• Types of approaches to preventing programme drop-out.• Any changes that could be made to the programme to help retain

parents/children?• Knowledge of changes for children/parents who completed the

programme.• Where do families tend to be six months later? Views on longer-

term impact of Family SMILES on children/parents.

6. Evaluation measures

• Experience of using the measures – benefits and challenges.


Interview Schedule for Family SMILES referrers

Background:

• Did they make the referral to Family SMILES themselves?

• Did they work with the family throughout their period of involvement in Family SMILES?

• Is the case still open?

• How many other cases referred to Family SMILES in the past?

Reason for referral to NSPCC

• Why did they think it would be a useful programme for the family to access?

• What did they hope the family would get out of participating in the programme?

– Improved protective parenting

– Confidence of child

– Improved ability of child to keep themselves safe

– Change in child protection status.

View on the programme

• Did it address the issues that they expected it to address?

• Thoughts on the structure and content of the programme – group work, parent work, joint work.

• Whether the programme filled any gaps in their own work.

• Whether there is anything missing from the programme that would support their work with the family more.

Contact with the NSPCC

• Level of contact with the NSPCC over the course of the families’ involvement with the Family SMILES programme:

– Updates on changes for the family, with regard to level of participation in the programme; passing on any concerns that arose during the course of the work; communicating positive changes during the course of the programme.

– Level of contact felt right?

• Time it took to allocate children to a group.


Longer-term changes

• Check if still involved in the case.

• Change for child/children since taking part in the programme:

– ability to stay safe; understanding of parent’s condition; change in confidence/self-esteem

• Change for parent/s since taking part in the programme?

– Protective parenting knowledge; understanding impact of mental health/substance misuse on their child.

• Change in relationships within the family since taking part?

• Change in risk levels for parent/s since taking part in the programme?

• Change in relationship between the family and Children’s Services since taking part?

– Probe whether any changes might be attributed to the programme.

• Check if they ever refer to any other similar services in the local area – how does Family SMILES compare?

• Are there any improvements that could be made to the programme?

• Any other comments or suggestions?


Appendix 3: Ethics OverviewAll NSPCC practitioners delivering Family SMILES attended evaluation training in order to gain an understanding of how the evaluation worked and the key ethical considerations in carrying out the evaluation. The ethical issues central to this evaluation are listed below:

• Gaining informed consent from service users – practitioners explained the purpose of the evaluation to service users, their role within the evaluation and the way in which information that they shared with the evaluation team would be used. Service users consented to the evaluation with a clear understanding of these issues. Consent forms were signed pre- and post-programme by parents who were happy to take part, and/or for their child to take part, in the evaluation. Verbal consent was obtained from children regarding their participation in the evaluation. The same process was also used for those parents and children who were part of the comparison group.

• Data protection, security and confidentiality – it was explained to service users that all information that they shared with the evaluation team would be treated in the strictest confidence unless they shared something that raised concerns about a child’s safety. The information sheet explained that the data would be held securely and how it would be used: i.e., to inform analysis and for reporting.

• Option to withdraw from study – service users understood at the time that they consented to be part of the evaluation that they could change their mind at any time and withdraw their involvement or contribution to the evaluation before the final report was produced.

• Protection of participants – Practitioners were requested to use their own judgement when asking service users to take part in the evaluation, particularly in relation to their capacity to cope with completing certain measures, to ensure that doing so did not cause them any harm. This was especially crucial when administering the CAPI, which is a lengthy measure containing some difficult and personal questions that may cause parents to reflect on their lives in such a way that causes them a level of distress. Practitioners were able to use their own judgement on where the CAPI should not be completed; this was particularly important when working with parents with mental illness or with drug or alcohol misuse problems. Similarly, practitioners were able to use their judgement to ensure that service users who may have felt vulnerable in an interview situation were not included in the sample for qualitative interviewing. In this way, participants were protected from any


potential harm. All interviews were gently wound down at the end to ensure that participants were left feeling as upbeat as possible at the end of the interview.

• Advice/support for participants – If an evaluation participant felt in need of extra support following an interview, researchers were able to suggest services that might be able to help. While practitioners completed the evaluation measures with service users, they were able to discuss any feelings that this process had brought up, and obtain advice and support directly from their NSPCC worker.

• De-briefing – All interviews and completion of measures ended with a discussion to explore how the service user experienced taking part in the evaluation, to ensure that they felt comfortable with what they had just done, and to provide them with the opportunity to ask any questions that they may have had.


Appendix 4: Standardised measures

The Strengths and Difficulty Questionnaire (SDQ)

• The SDQ measures the emotional and behavioural problems of children and young people as well as their strengths.

• It has been used in the Framework for the Assessment of Children in Need and their Families (Department of Health et al, 2000) and by CAMHS Outcome Research Consortium (CORC).

• Young people aged 11 years or over, and younger children who practitioners felt were able to complete the SDQ alone, were offered the self-complete version of the questionnaire. The SDQ was completed by the parent/carer if the child was younger than 11 or unable to complete it themselves.

• It contains four difficulty subscales: Hyperactivity, conduct problems, emotional symptoms and peer problems, plus a strength subscale of pro-social items.

• The higher the total score, the higher the level of difficulty.

• There are four scoring thresholds for the total scores and subscale scores: Very high, high, slightly raised and close to average.

• For the purposes of understanding clinical change, the two lower bands and two higher bands have been grouped together and have been defined as non-clinical or clinical difficulty, respectively.

HoNOSCA (Health of the Nation Outcome Scales Child and Adolescent mental health)

• The HoNOSCA captures practitioners’ perspectives on children’s behavioural and emotional difficulties.

• It measures the range of physical, personal and social problems associated with mental health.

• Practitioners provide a score between 0 and 4 for each of 13 criteria set out in the HoNOSCA covering four broad categories: behavioural problems; impairment; symptomatic problems; and social problems.

• There are no clinical cut-off points for HoNOSCA, rather it allows for a change in average scoring to be identified over the course of the programme.


Rosenberg Self-esteem Scale (Adapted)

• The self-esteem questionnaire is based on the Rosenberg self-esteem questionnaire, adapted by the NSPCC for use with children.

• Since it has been adapted, it is not standardised and therefore no clinical thresholds are available.

• It is relatively short (10 statements) and includes reversed scoring.

• Total scores range from 0 – 30, with a higher score indicating a higher level of self-esteem

Child Abuse Potential Inventory

• Developed in the 1970s by Joel Milner to assist social services in the US to screen parents who ‘potentially’ may be at risk of physically abusing their child. Subsequently it has been used as an assessment tool and for evaluation purposes.

• Used for Family SMILES as an evaluation tool only and not for predicting abuse. It is a reliable tool for measuring change in protective parenting.

• Validated to determine six key aspects:

– Attitude: rigidity; distress; unhappiness

– Interpersonal: problems with child and self; problems with family; problems with others.

• Also includes scales to measure ego strength and loneliness.

• Inbuilt validity scales to filter out parents who may be faking good responses or being inconsistent or random in their responses.

• More than 1,000 journal articles, chapters, books, dissertations, theses, convention papers, and unpublished reports describing the psychometric characteristics and/or applications and/or limitations of the CAPI.

• 160-item questionnaire. Cut-off score of 215 indicates that parents with scores above this threshold have poor protective parenting behaviours.

Evaluation wheels

• Evaluation wheels are completed by parents and children.

• They are not standardised measures.

• Respondents rate themselves between 1 and 5 (1 being low, 5 being high) against 5 criteria (parents) and 6 criteria (children), such as confidence in being able to do the best for their children (parents’ Wheel) and being able to talk to parents about their drug or alcohol misuse (children’s Wheel).


Appendix 5: Characteristics of children and parents at each point of the referral journey and the evaluation

Characteristics of parents

Stage of case Characteristics of parents/carers

Point of referral (N=677) Gender of parents/carers engaged

567 female, 93 male, 17 unknown

Start of assessment (N=502) Gender of parents/carers engaged


Start of the intervention (N=365)

Gender of parents/carers engaged


End of Family SMILES (not full programme) (N=90)



End of Family SMILES (full programme) (N=191)



Stage of evaluation Characteristics of parents/carers

All at T1 (N=188)

Of which valid for analysis (N=168)

Gender of parents/carers 146 female, 15 male, 27 unknown

Gender of parents/carers for valid completions only


Pre- and post-programme (N=74)

Of which valid for analysis (N= 59)

Gender of parents/carers 63 female, 7 male, 4 unknown

Gender of parents/carers for valid completions only


Pre-, post- and 6-month follow up after programme (N=22)

Of which valid for analysis (N= 18)

Gender of parent/carer 20 female, 1 male, 1 unknown

Gender of parent/carer for valid completions only



Characteristics of children

Stage of case Characteristics of children

Point of referral (N=679)

Gender 319 girls, 341 boys, 19 not recorded

Age Under 8: 35; 8–9 years: 194; 10–11 years: 229; 12–13 years: 149; over 13: 65; not recorded: 7

Start of assessment (N=574)



Start of the intervention (N=472)


Age Under 8: 30, 8–9 years: 146; 10–11 years: 158; 12–13 years: 94; over 13: 37; not recorded: 4

End of Family SMILES (not full programme) (N=75)


Age Under 8: 2; 8–9 years: 23; 10–11 years: 24; 12–13 years: 18; over 13: 8

End of Family SMILES (full programme) (N=333)



Stage of evaluation

Characteristics of children

T0 only (n=65) Gender 16 female, 15 male, 34 not recorded

Age 8– 9 years: 7; 10–11 years: 15; 12–13 years: 6; over 13: 3 not recorded: 34

T0 and T1 (n=31) Gender 14 female, 12 male, 5 not recorded

Age 8–9 years: 6; 10–11 years: 11; 12–13 years: 5; over 13: 3; not recorded: 5

T1 only (N=463) Gender 156 females, 137 males, 170 not recorded


T1 and T2 (N=199)

Gender 93 females, 75 males, 31 not recorded

Age Under 8: 15, 8–9 years: 52; 10–11 years: 66; 12–13 years: 30; over 13: 6; 30 not recorded

T1, T2 and T3 (N=65)

Gender 37 females, 22 males, 6 not recorded

Age Under 8: 7; 8–9 years: 16; 10–11 years: 24; 12–13 years: 10; Over 13: 2; not recorded: 6


Appendix 6: Statistical analysis and qualitative data managementExcel was used to carry out 1-tailed T-Tests to look at change in mean scores between the beginning and end of FED UP. SPSS was used to carry out the McNemar test – for proportional movement between the clinical and non-clinical threshold – and the Repeated Measures ANOVA and Friedman’s 2-way ANOVA, to test the changes in mean scores across three time points: pre- and post-programme and at six-month follow up.

I. Strengths and Difficulties Questionnaire data: Change in mean score, pre- and post-Family SMILES based on the 1-tailed T-Test (n=199)

SDQ subscale Mean at T1 Standard Dev. at T1

Mean at T2 Standard Dev. at T2

P-Value

Emotional difficulties

4.37 2.29 4.14 2.30 0.08

Conduct problems

3.34 2.14 3.02 1.83 0.004*

Hyperactivity 4.87 2.12 4.52 2.11 0.006*

Peer problems 3.95 2.20 3.80 2.32 0.15

Pro-social 8.04 1.99 8.22 1.65 0.05*

Total score 16.53 5.92 15.48 5.84 0.001*

* Statistically significant

II. (a) Strengths and Difficulties Questionnaire data: Proportional shift in children from a clinical level of difficulty (high and very high) to a non-clinical level (close to average and slightly raised) between the beginning and end of Family SMILES. Based on an Exact McNemar’s Test (n=199)

Level of difficulties Pre-programme (per cent)

Post-programme (per cent)

P-value

Non-clinical range (close to average and slightly raised bands)

56% 64% 0.46

Clinical range (high and very high bands)

44% 36%


II. (b) Strengths and Difficulties Questionnaire data: Movement of children from a level of clinical need to non-clinical need (and vice versa) at the end of Family SMILES (n=199)

Clinical level of need post-Family SMILES

Non-clinical level of need post-Family SMILES

Clinical level of need pre-Family SMILES

55 32

Non-clinical level of need pre-Family SMILES

17 95

III. rosenberg Self-Esteem Questionnaire Adapted for Children: Change in mean score, pre- and post-Family SMILES, analysis based on the 1-tailed T-Test (n=209)



P-Value

Total score 18.45 5.62 19.45 5.53 0.001*

*Statistically significant

Iv. HoNOSCA: Change in mean score, pre and post Family SMILES, analysis based on the 1 tailed T-Test (n=)



P-Value

Total score 7.62 5.34 6.14 4.70 0.0000*


v. CAPI: Change in mean scores, pre- and post-Family SMILES, analysis based on the 1-tailed T-Test (n=59)

Mean at T1

Standard Dev. at T1

Mean at T2

Standard Dev. at T2

P-Value

Distress 190.29 59.54 169.75 72.30 0.0002*

Rigidity 14.36 14.83 13.81 13.75 0.31

Unhappiness 38.39 18.26 30.63 18.27 0.000*

Problems with child 8.76 8.58 6.71 7.29 0.006*

Problems with family 18.32 13.12 15.75 14.26 0.07

Problems with others 17.05 6.85 16.6 6.13 0.26

Total score 287.17 88.85 253.42 100.39 0.000*

Ego strength 9.27 8.24 12.47 10.23 0.000*

Loneliness 12.02 2.84 11.36 3.78 0.03*



vI. Parents’ evaluation wheel score, pre- and post-Family SMILES, analysis based on the 1-tailed T-Test (n = 74)

Mean at T1

Standard Dev. at T1

Mean at T2

Standard Dev. at T2

P-Value

How much I think my child is affected by my behaviour.

3.88 1.18 3.92 1.04 0.40

How confident I feel that I am doing the best I can for my child.

3.53 1.09 4.03 0.99 0.000*


3.48 1.21 3.95 1.07 0.001*

How confident I feel in asking for help when I need it.

3.30 1.38 3.77 1.23 0.004*

How much knowledge I have about children’s needs at different stages of their development.

3.78 0.89 4.10 0.78 0.007*


vII. Children’s evaluation wheel score, pre- and post-Family SMILES, analysis based on the 1-tailed T-Test (n =230)

Mean at T1

Standard Dev. at T1

Mean at T2

Standard Dev. at T2

P-Value

I can talk to someone if I’m worried about my parent’s health.

3.59 1.33 4.06 1.19 0.000

I can talk to my parent about how their mental health affects me.

3.12 1.46 3.49 1.42 0.001

I can easily make friends 3.81 1.29 3.85 1.26 0.30

I am able to have fun when I want to.

3.93 1.23 3.88 1.21 0.32

I feel supported by others around me.

3.99 1.12 4.15 1.05 0.03


vIII. rosenberg Self-Esteem Questionnaire Adapted for Children: Change in mean score, pre-, post- and six months after Family SMILES, analysis based on the repeated measures ANOvA (n=69)

Descriptive Statistics

Mean Std. Deviation N

Rosenberg T1 Total score SMILES 18.39 6.360 69



Tests of Within-Subjects Effects

Measure: Rosenberg total

Source Type III Sum of Squares

df Mean Square

F Sig.

Rosenberg Smiles

Sphericity Assumed 81.768 2 40.884 2.808 .064

Greenhouse-Geisser 81.768 1.821 44.914 2.808 .069

Huynh-Feldt 81.768 1.868 43.775 2.808 .068

Lower-bound 81.768 1.000 81.768 2.808 .098

Error (Rosenberg Smiles)

Sphericity Assumed 1980.232 136 14.561

Greenhouse-Geisser 1980.232 123.798 15.996

Huynh-Feldt 1980.232 127.019 15.590

Lower-bound 1980.232 68.000 29.121

Ix. Strengths and Difficulties Questionnaire data: Change in mean score, pre-, post-and six months after Family SMILES, based on the repeated measures ANOvA (n=65)

Descriptive Statistics

Mean Std. Deviation N

SDQ T1 score SMILES 16.23 6.703 65



Tests of Within-Subjects Effects

Measure: SDQ total

Source Type III Sum of Squares

df Mean Square

F Sig.

SDQ Sphericity Assumed 48.810 2 24.405 1.598 .206

Greenhouse-Geisser 48.810 1.863 26.197 1.598 .208

Huynh-Feldt 48.810 1.917 25.463 1.598 .207

Lower-bound 48.810 1.000 48.810 1.598 .211

Error (SDQ)

Sphericity Assumed 1954.523 128 15.270

Greenhouse-Geisser 1954.523 119.246 16.391

Huynh-Feldt 1954.523 122.681 15.932

Lower-bound 1954.523 64.000 30.539


Framework themes and sub-themes Family SMILES interviewsFramework in NVivo was used to analyse interview transcripts. Data was classified and organised in the analytical frameworks according to key themes, concepts and categories.

Framework for children and parent interviews

1.0 Analysis of standardised measure

2.0 Practitioner case closure summary

3.0 Family environment

4.0 Changes after or during programme

4.1 Talking about the issue and impact

4.2 Knowledge or confidence in dealing with crisis

4.3 Change in own behaviour and impact

4.4 Change in other behaviour and impact

4.5 Change in relationship and impact

4.6 Change in family situation and impact

4.7 Change in aspiration, motivation, hopefulness

4.7 Change in understanding about issues

4.8 Change in using other service

5.0 Role of the programme

5.1 Activities that helped or not

5.2 Practitioner support

5.3 Peer support (for children)

5.4 Support plan

5.5 External factors

5.6 Expectations met or not

6.0 Suggestions

7.0 Learning

8.0 Other


Framework for Family SMILES practitioner interviews

1.0 Nature of positive change observed

1.1 Change for the child

1.2 Change for the parent

1.3 Change for the family

2.0 Reasons for the positive change

2.1 Programme-related elements

2.2 Non-programme-related elements

3.0 Nature of negative change

3.1 For the child

3.2 For the parent

3.3 For the family

4.0 Reasons for negative, little or no change

4.1 Programme-related elements

4.2 Non-programme-related elements

5.0 Aspects that worked well

5.1 Assessment

5.2 Group work

5.3 Parent work

5.4 Safety plan

6.0 Aspects that did not work so well

6.1 Assessment

6.2 Group work

6.3 Individual work with parent

6.4 Safety plan

7.0 Problems implementing the model

8.0 Skills needed to deliver programme

9.0 Experience of using standardised measures

10.0 Sustained change


Framework for referrer interviews

1.0 Referral experience

1.1 Referral process

1.2 Reasons for referral

1.3 Fit with referrers workload

1.4 Availability of similar services

2.0 Positive changed observed

2.1 For children

2.2 For parents

2.3 For family

3.0 Reasons for positive change

3.1 Programme-related

3.2 Non-programme-related

4.0 No change or negative observed

4.1 For children

4.2 For parents

4.3 For family

5.0 Reasons for no change in negative cases

5.1 Programme-related

5.2 Non-programme-related

6.0 Aspects that worked well

6.1 Children’s group work

6.2 Individual parent work

6.3 Joint work

7.1 Aspects that did not work so well

7.1 Children’s group work

7.2 Individual parent work

7.3 Joint work

8.0 Engagement with the NSPCC

8.1 Contact during programme

8.2 Overall relationship

8.3 Involvement in external or child protection discussions

9.0 Suggestions.

www.nspcc.org.ukregistered charity numbers 216401 and sc037717.

http://www.nspcc.org.uk

Documents

BuildiNg childreN’s coNfideNce aNd improviNg pareNts’ … · 2020-01-09 · Children valued the opportunity to meet with peers facing similar difficulties to themselves, and learning