Embed Size (px)
Citation preview
Child Protection Initiative
Measuring the hard to measure:
Global child protection indicators
Paper presented at the
2013 International Society for Child Indicators Conference
Seoul, South Korea
May 30, 2013
AUTHORS:
Ms. Meri Ghorkhmazyan
Child Protection Initiative
Monitoring and Evaluation Advisor
Save the Children
Ms. Denise Stuckenbruck
Child Protection Initiative
Global Programme Manager
Save the Children
PRESENTERS:
Ms. Meri Ghorkhmazyan
Child Protection Initiative
Monitoring and Evaluation Advisor
Save the Children
Ms. Shana Peiffer
Child Protection Senior Specialist, Asia Region
Save the Children US
To contact Save the Children’s global Child Protection Monitoring and Evaluation team, please write
2
TABLE OF CONTENTS
1. INTRODUCTION ........................................................................................ 3
2. QUALITY OF SERVICES INDICATOR .................................................. 5
2.1. Brief history of the indicator management process ...................................................... 5
2.2. The tool and methodology............................................................................................ 7
2.2.1. The structure of the tool ....................................................................................... 7
2.2.2. How the tool should be applied ......................................................................... 13
3. USE OF THE DATA .................................................................................. 14
3.1. Indicator and standard level ....................................................................................... 14
3.2. Service provider type ................................................................................................. 15
3.3. Country, regional and global levels............................................................................ 16
4. LIMITATIONS AND CHALLENGES .................................................... 16
4.1. Selecting only new or only experienced service providers every year ...................... 16
4.2. Failing the standard .................................................................................................... 17
4.3. Irrelevant standards .................................................................................................... 18
4.4. Omitted components and categories when filling out the tool ................................... 18
4.5. Capacity of the Assessors ........................................................................................... 19
4.6. Excel database ............................................................................................................ 19
5. WAY FORWARD IN MEASURING IMPACT ...................................... 19
6. ANNEXES: Standards and indicators per service provider .................. 22
6.1. Formal Service Providers ........................................................................................... 22
6.2. Community Based (Informal) Providers .................................................................... 25
3
1. INTRODUCTION
Child protection is a relatively new sector in international development and, as a result, lags
behind other sectors, such as health and education, in terms of the evidence base and
methodologies for generating evidence. Good progress has been made in the development of
standards and guidelines in some areas, such as international good practice guidelines for
work with unaccompanied and separated children, e.g. Best Interest Determination tool,
International Guidelines on Alternative Care for Children, and for gender based violence, yet
a significant gap remains with the existence of few internationally validated indicators for
child protection that measure effectiveness of prevention and response interventions
addressing abuse, neglect, exploitation and violence affecting children.
The child protection sector faces significant methodological challenges in developing
indicators that can be applied across a range of cultures and contexts. Many child protection
issues are hidden and sensitive, and are, therefore, missed in routine data collections or
population-based household surveys. Often child protection concepts can be subjective and
hard to measure through standard indicators, tools and approaches. Furthermore, the capacity
of researchers and practitioners in many countries to develop and use child protection
indicators and methodologies is limited.
Against this context, Save the Children has developed, over the last three years, a small set of
global child protection outcome indicators to monitor the organisation’s progress in this
important area. These indicators are designed to be used across a range of child protection
interventions, and across cultures and contexts. The indicators will be rolled out across 80
Save the Children country offices in 2013.
This paper describes how Save the Children uses one of the global child protection indicators
to effectively measure the quality of child protection services that children, their families and
4
caregivers receive through the organization’s support. It provides the technical definition of
the indicator, as well as the tools and methodology that Save the Children has developed to
measure it. This paper also provides a brief summary of the process that Save the Children
implemented to test and improve the indicator.
Save the Children defines Child Protection as measures and structures to prevent and respond
to abuse, neglect, exploitation and violence affecting children. In 2012, 65 Save the Children
country programmes reported implementing child protection programmes, directly reaching
about 2.4 million children, with an annual budget of approximately 160 million USD. Those
programmes aim to improve the psycho-social wellbeing of children, strengthen the family
environment through positive parenting, enable families to stay together, carry-out tracing
and reunification, and to minimise the risks that children in various contexts are exposed to
(harmful child labour, sexual abuse and exploitation, etc.). Wherever possible, to ensure the
sustainability of the results achieved through its programmes, Save the Children partners with
local authorities, non-governmental organisations and community groups. In certain contexts,
especially during humanitarian crises or in fragile States, Save the Children often delivers the
services itself, while it works to build local capacity in the long-term.
As one of the world’s largest organisations working in child protection internationally, Save
the Children’s experience will demonstrate some of the achievements, challenges and lessons
in developing global indicators that aim to monitor a number of the hard to measure issues of
child protection. The lessons learned from this process, as well as the indicator itself, can be
replicated and used by other like-minded organisations.
5
2. QUALITY OF SERVICES INDICATOR
2.1. Brief history of the indicator management process
In 2010, as part of the process of unifying its international programs, Save the Children
decided that it would, for the first time in its history, design and implement a new system of
global indicator monitoring across all its country programmes. As a result, a number of global
indicators were developed and piloted in the last three years. The Quality of Services Child
Protection Global Outcome Indicator was one of the indicators that underwent a thorough
piloting and testing process to determine:
The feasibility of using the indicator as a measure of Save the Children’s child
protection work;
The challenges associated with implementation of the monitoring tools;
The validity and reliability of the tools and reporting systems; and
Any additional technical support and capacity building/training that may be needed to
guide programme staff in data collection and reporting.
The rationale for measuring the quality of services is that solely improving access to child
protection services may not benefit children and families unless those services are of a
minimum quality standard (defined internationally or internally, and documented in the
programme strategy or plan). This indicator was designed to help assess whether the services
that Save the Children supports are of a minimum quality standard, and therefore likely to
benefit the users.
In 2010-2011 the indicator was piloted in Nepal and the Philippines and, in 2012 in Albania,
Liberia, Senegal, Tanzania and Zimbabwe. The feedback received during the pilot phase
informed the improvement of the technical guidance and the tools used to monitor progress
6
against the Quality of Services indicator, to ensure that they are culturally sensitive and
relevant to the various contexts where they are applied.
Overall, the piloting of the Quality of Services indicator was very successful in all countries.
The teams participating in the pilot found that the tool generated important evidence to track
progress of the quality of our work, was feasible to administer, and provided useful
information to improve programme quality. In particular, the teams reported that, if managed
appropriately, it helped to build partnerships with service providers and strengthen capacity
around child protection issues. As a result, the service providers understood the importance of
delivering quality services and shared the ambition of meeting as many of the quality
standards as possible.
Table 1: The Quality of Services Child Protection Outcome Indicator definition
Type of indicator Global Outcome Indicator
Reporting All Save the Children country offices with relevant child protection programmes
Definition (what is
this indicator?)
% of prevention and response interventions supported by Save the Children,
which meet quality standards.
Numerator
Denominator
# of prevention and response interventions supported by Save the Children
which meet quality standards
Total number or prevention and response interventions assessed
What information
do country offices
need to provide
Country offices should submit the complete dataset collected for this indicator,
using the provided Excel Management Information System or any other
preferred Management Information System. Country offices should also
complete the summary reporting form which requires the numerator and
denominator for the indicator, disaggregated by type of service and sub-theme.
Country offices also need to provide a summary of the methodology used,
including sample size (if sampling used), date of assessments and description of
approach.
How will this
information be used
The information provided by country offices will be used to undertake global
analysis in service quality. At country office level, based on the information
gathered, improvement plans can be developed with the service providers to
promote better services by improving the jointly identified priority areas.
Data source(s) Assessment of service providers drawing on interviews with service staff,
children and families who use the service.
Timing of data
collection
At least once a year. All assessments should take place at roughly the same time
of year, e.g. all assessments to be completed in one month of the year.
7
2.2. The tool and methodology
2.2.1. The structure of the tool
To ensure adequate roll out of the Child Protection Quality of Services indicator, a detailed
handbook was developed by Save the Children, providing guidance, tools and additional
resources on how to collect, collate, analyse and use the data for improvement and decision-
making.
The quality of the provided services is determined through an assessment and scoring
process, which is carried out against a set of quality standards using a scale ranging from 1-4.
While there are standards for particular types of work with children1, child protection as a
sector does not currently have standards that measure the quality of services provided to
children, their families and caregivers. For the purpose of monitoring this global outcome
indicator, a set of generic standards that all services and community based mechanisms
working in child protection should meet has been developed2. These standards and associated
indicators are based upon a literature review across agencies in the child protection sector.
Save the Children works with a wide range of child protection service providers, and
acknowledges that the standards which a government-run service would be expected to meet
might vary from those run by a voluntary community group. Therefore, this indicator collects
data based on two categories: (a) Formal service providers, and (b) Community-based
(informal) service providers. As a result, there are two different versions of the
1 For example, there are: international good practice guidelines for work with unaccompanied and separated
children; UN Guidelines on Alternative Care for Children; and standards are being drafted for all child
protection interventions in emergencies. Similarly in some countries, there are national standards for residential
care institutions, for gender based violence (GBV) and other child protection services. 2 The tool recommended to use for measuring this indicator, does not include standards which are specific to
certain types of services or may be relevant in particular countries only. However, additional specific standards
which are relevant for a country context or particular service may be added into the tools by the country office
as appropriate.
8
accompanying assessment tool, each tailored to the relevant type of service provider. For
formal service providers, the monitoring tool contains 11 standards with 25 related indicators
and for community-based (informal) providers the monitoring tool contains 10 minimum
standards with 23 related indicators. While it is recommended that country offices using the
tool follow the given standards and the indicators to ensure that data collected is compatible
and can be globally aggregated, a certain level of flexibility to adapt the tool is possible to
ensure that standards are relevant in the country context. The standards that the tool aims to
assess are listed in Table 2.
Table 2: Standards assessed by the indicator tool
Formal service providers Community based (informal) service providers
1. Information about the service 1. Information about the community mechanism
2. Child safeguarding 2. Roles and responsibilities
3. Compliance with national regulations 3. Community ownership and representation
4. Equality and non-discrimination 4. Child safeguarding
5. Child participation 5. Equality and non-discrimination
6. Staff experience, training and support 6. Child participation
7. Placement, planning and review 7. Staff experience, training and support
8. Privacy 8. Referral
9. Case management 9. Records
10. Records 10. Effectiveness
11. Effectiveness
Each of the standards has a set of indicators designed around subcategories of the standard.
For example, the child safeguarding standard for formal service providers has the following
indicators:
A child safeguarding policy and/or Code of Conduct is in place for service staff and
users;
The service provider has a mechanism in place for receiving and responding to
complaints from children;
9
All staff have been screened before hiring for suitability to work with children and in
line with any national regulatory requirements;
Service provider maintains an appropriate adult/child ratio (this well depend on type
of service and any national regulatory requirements.
The full list of standards and indicators per service provider are available in the Annex.
During the assessment of the standard, each of those indicators is assessed separately and is
given a score of 1 to 4, where:
1 = Indicator is not achieved: There is no evidence that any efforts have been made to
achieve the monitoring indicator;
2 = Indicator is almost achieved: Some efforts to achieve the monitoring indicator are
observed, although they are inconsistent. Some additional work is required to ensure
consistency;
3 = Indicator is achieved: There is consistent evidence that the monitoring indicator
has been successfully reached in the service/service provider;
4 = Indicator is exceeded: There is direct and consistent evidence that the indicator
has been achieved. In addition, there is evidence that a variety of methods is used to
go beyond the minimum expectation and ensure an exceptionally high level of quality
during implementation;
NA = Indicator is not relevant for service being assessed.
A definition of what the scores 1-4 mean for each indicator is provided as shown below in
Table 3. 3
3 The most updated scoring sheet can be requested from Save the Children by sending an e-mail to
10
Table 3: Example of the scoring sheet (Community based service providers)
Indicator Ratings Not
applicable
1 2 3 4
Standard 4: Child safeguarding
4.1 Community
(informal)
provider has a
child safeguarding
policy and/or
Code of Conduct
(CoC) in place
NA Policy &/or
CoC is not
in place
Policy &/or
CoC is in place
but weak with
few members
signing
Policy &/or
CoC is in place
and all
members have
signed
Policy &/or CoC
is in place,
members have
signed and there
is good
knowledge and
compliance
4.2 A mechanism
exists for
receiving and
responding to
complaints that
children know
how to access
NA No
complaints
mechanism
in place
Complaints
mechanism in
place but is not
functioning or
child-friendly
Complaints
mechanism is
in place which
is functioning
and child-
friendly
Excellent
complaints
mechanism in
place which is
highly
functioning, and
child-friendly
An Excel-based tool is used to document the scores and determine the level of achievement
of an indicator against a quality standard.
The extent to which each individual quality standard has been met by the service providers is
decided by mapping the final results of the scoring against one of the three pre-determined
colour-coded RAG (red, amber or green) achievement levels. Once each indicator is given a
score, the average score of all indicators is calculated and visually represented by the colour-
coded achievement level, where values 1.0-1.9 are in category red (standard not at all
achieved), values 2.0-2.9 are in category amber (standard almost achieved) and values 3.0
and above in category green (standard achieved or exceeded), as shown in Table 4.
11
Table 4: Quality of services assessment sheet part 1: assessing each standard separately
Standard 1 Standard 2
# Background information about the service 1.1 STD1_R
ATING 2.1 2.2 2.3 2.4
STD2_R
ATING
1
Project: Victim Friendly Court System
3 3 1 2 3 3 2.25
Location: Harare
Date: 14/11/2012
Type of service: Prevention and response
Type of service provider: Government
Type of Save the Children support: More
than 1 type
Duration of Save the Children support:
More than 2 years
To assess the extent to which all quality standards are met by each service provider, the
average grade of all indicators is calculated and mapped against the achievement level,
following the approach explained above, placing the service provider’s (SP) achievement in
the red, amber or green category, as demonstrated in Table 5.
Table 5: Quality of services assessment sheet part 2: summarizing the rating assessment
#
Proje
ct
name
ST
D1
_R
AT
IN
G
ST
D2
_R
AT
IN
G
ST
D3
_R
AT
IN
G
ST
D4
_R
AT
IN
G
ST
D5
_R
AT
IN
G
ST
D6
_R
AT
IN
G
ST
D7
_R
AT
IN
G
ST
D8
_R
AT
IN
G
ST
D9
RA
TI
NG
ST
D1
0_
RA
TI
NG
OV
ER
AL
L
RA
TI
NG
RATING
SUMMARY
1 SP. 1 3 2.3 2 2 2 2 2 2 2 2 2.1 red 1
2 SP. 2 3 3 3 3 3 3 3 3 3 3 3.0 amber 3
3 SP.3 2 2.5 1.5 2 2 2 2 2 2 2.3 2.1 green 1
4 SP. 4 2 2.7 1 2 n/a n/a n/a n/a n/a 1 1.9 Total
# of
assess
ed
servic
es 5 5 SP. 5 2 2.7 1 2 n/a
n/a n/a n/a n/a
3 2.1
12
This assessment is conducted separately for formal and informal service providers and results
are recorded in separate Excel spreadsheets, which then are summarized to demonstrate the
progress at a country office level, as shown in Table 6.
Table 6: Quality of services assessment sheet part 3: consolidation of assessment results
Summary
data
Number
of
services
assessed
Number
of
services
per
category
% services
per category
Service providers (formal, NGO) RED 4 0 0%
AMBER 3 75%
GREEN 1 25%
Community based child protection
mechanisms
RED 5 1 20%
AMBER 3 60%
GREEN 1 20%
Global Outcome Indicator: % of prevention
and response service providers supported by
Save the Children during a 12-month period
which meet quality standards
RED 9 1 11%
AMBER 6 67%
GREEN 2 22%
In this three-level RAG rating scheme, only service providers that have achieved an average
score of 3 and above will be rated as fully meeting the quality standards. Service providers in
the red category need significant attention since they probably haven’t achieved even the
fundamental components of the standards. It is expected that the transition from red to amber
at the standard level may be very quick, whereas the transition from amber to green may
require longer time and more capacity building.
13
Once country office data is received, Save the Children aggregates all the results at the global
level using the same Excel tool, adding one more level to track data by country office.
2.2.2. How the tool should be applied
The scoring, in other words, the assessment of the quality of the services, is a consultative
process led by an experienced facilitator who is familiar with the context, programmes, and
the child protection sector. S/he can be either from within Save the Children or the partner
organisation, or an external consultant. Multiple interviews and focus group discussions are
held with service providers and users of the service, and the findings are triangulated with
secondary data (including policy documents, workplans etc.), observations, and other data
available or are verified through a third source to check the validity of the findings. At the
end of the assessment the ratings are discussed and mutually agreed upon. Workplans for the
coming year are developed based on the identified areas for improvement.
All relevant country offices are required to conduct the assessment of the quality of services
on an annual basis. Once the baseline for a particular service is in place and areas for
improvement are identified, in the follow-up years the country offices need to make sure that
the standards that were met are still maintained, and the improvements planned have
occurred.
Management of this indicator requires substantial capacity strengthening and planning. It
starts with identification of the relevant service providers and ensuring that a minimum
sample size of relevant services is maintained (15 formal or services community-based
services). If Save the Children is supporting less than 15 child protection services, the country
offices are required to include all services and not a sample. Once the services to be assessed
are identified, preparation of the data collecting team and tools follows alongside the
14
translation and training of teams, where required. The assessment may take between 2-3
hours to a full day with each service provider.
3. USE OF THE DATA
The data collected through the Quality of Services indicator can be used to draw findings and
conclusions at indicator, standard, service provider, country, regional and global levels. Once
the data is compiled, the trends are fairly easy to notice. Below are some of the examples of
the types of findings and conclusions that can be drawn from the Quality of Services
indicator data.
3.1. Indicator and standard level
Analysing the achievement of the standards and their relevant indicators across country
offices can demonstrate whether there are any particular indicators and/or standards that need
significant improvement. For instance, in Table 3, the data shows that at least three out of
five assessed service providers have not met the standard. If this were the assessment for
formal service providers, the standard is about compliance with national regulations. Not
meeting the standard means that the service provider was unable to demonstrate evidence of
being officially accredited4. Therefore, the conclusion of the assessment is that three out of
five service providers are not accredited. This information can be used by Save the Children
to understand why the service providers have not yet been accredited, and what can be done
to support them to reach accreditation.
On the other hand, the standards that are fully met can be analysed to document the
interventions and methods that have been successfully used by a particular country
4 The indicator of the standard is: 3.1 Service provider has acquired relevant government licensing or
accreditation
15
programme. This knowledge can be shared as a best practice with the service providers who,
in their turn, can adapt and adopt available tools to improve their effectiveness.
At the global level Save the Children can track the standards that have not been and continue
not being met by many country offices and analyse what causes the challenges. This analysis
might reveal a need to make systemic corrections per se. For example, if the global analysis
demonstrates that most of both formal and informal service providers struggle to meet the
safeguarding standard, further analysis of the standard’s indicator assessment may show that
those service providers simply do not have a policy or code of conduct in place. In this case
Save the Children may decide to carry-out workshops, webinars and other activities to refresh
staff members’ knowledge of the Child Safeguarding policy and Code of Conduct, who then
can share it with the service providers for adaptation and adoption.
As data is collected over the years, Save the Children may be able to track not only progress
but also whether there are any standards that have not improved, or have decreased in their
rating over time. Also, if any particular service providers succeed in moving from “red” to
“green” in a short period of time on a particular standard, their performance might be studied
and documented as best practice.
3.2. Service provider type
The data can also be analysed to see whether there are any particular standards that are
achieved by formal service providers, which informal service providers struggle to achieve or
vice versa. This information can support country offices to encourage learning across service
providers, by facilitating sharing of best practice and solutions, working models and tools
across service providers.
16
3.3. Country, regional and global levels
A global overview of the Quality of Services data, viewed through the lens of service
provider and geography, can be used by the senior management of the organization to
determine where, and for what types of service providers, technical support and supervision
should be improved. Save the Children can also determine what support is required, which
geographic areas need the most support, and which lessons learned can be extracted and
shared with others that might benefit from it. The data could also help determine where and
when certain evaluations and reviews need to be undertaken to learn more about specific
challenges.
4. LIMITATIONS AND CHALLENGES
While the implementation of the Quality of Services indicator can be very informative and
useful, there are a number of challenges that need to be considered.
4.1. Selecting only new or only experienced service providers every year
On an annual basis country offices may be assessing services providers that are new to child
protection because of the timing of the project cycle or expansion of programme activities.
However, if every year the country offices assess many new service providers and are not
able to follow up on the previously assessed ones, the data collated and reported at the
country office level may be insufficient and irrelevant, particularly for tracking progress over
time. Since it is expected that new service providers will meet fewer of the standards than the
more experienced ones, the data reported annually for the country programme may show little
progress of achieving the standards over time.
17
Similarly, wanting to demonstrate success, country offices may be tempted to select the
experienced service providers with who the programme has had a long-term relationship and
has invested in capacity building and the quality of service delivery.
When aggregated regionally or globally this may also not present an accurate picture of the
work Save the Children is doing in building the capacity of the service providers. While we
understand that this limitation will probably not be fully addressed in the first period of
managing the indicator, Save the Children is hoping to improve its sampling framework over
the years so that the selection of the service providers is done uniformly across country
offices. Currently, the data management tool allows users to filter the service providers by the
length of relationship with Save the Children, which can be used to better analyse the
achievement of quality standards relative to length of capacity building efforts, and
demonstrate where progress is being made even if this factor is skewing the overall country
programme data. Additionally, country programmes are encouraged to select service
providers that are both experienced and new to the sector, focusing on those service providers
that can benefit from the assessment.
4.2. Failing the standard
The ultimate objective of the assessment is to demonstrate whether or not the services
provided meet the quality standards, and to what extent improvement has been made on those
that did not meet the standards. Because the bar is set high, most service providers are
initially discouraged by the final results. Additionally, service providers might be less willing
or transparent in the rating process, since they may mistakenly believe that the results of the
assessment might lead to decrease in funding if the standards have not been fully met. To
avoid this kind of disappointment, the person facilitating the assessment needs to put more
emphasis on the importance of demonstrating improvement over time than on passing or
18
failing the standard at the time of assessment. As mentioned in the description of the tool, it is
expected that moving from the red category to amber may be relatively easy compared to
moving from amber to green, although improvements may still occur. To help the service
providers demonstrate improvement, a numerical rating is also provided. For example, a
service provider might remain in the amber category for three years, but the overall rating
may increase from 2.2 to 2.4 in year 1, from 2.4. to 2.6 in year two and from 2.6 to 2.9 in year
three. This will allow the organization to track progress within the same category of rating,
and will allow the service provider to demonstrate what improvement has been made and
where.
4.3. Irrelevant standards
While the standards have been tested and piloted in a number of countries and regions, there
may be standards or definitions that are irrelevant for certain contexts. In those cases
standards have to be modified and made relevant for the country programme, but this might
affect the results when aggregated globally. These types of challenges have to be taken into
account when data is being collated and analysed at the global level. Save the Children
recommends that country offices do not change the standards or the indicators, and first
discuss with the global team should those changes be needed.
4.4. Omitted components and categories when filling out the tool
It is important that all standards and all indicators are measured. Sometimes country offices
might miss collecting data on one or more indicators, which later will skew the overall rating
of the assessment. It is important that during the collation of the data on this indicator, only
assessments that have complete datasets available are considered. Once country programmes
report the first version of the data, Save the Children’s technical monitoring and evaluation
19
advisors review the draft submitted and identify any reporting mistakes that might have
occurred, encouraging country teams to revise and correct their final reports.
4.5. Capacity of the Assessors
This assessment should be carried out as a consultative process, which requires an
experienced facilitator. The facilitator needs to be skilled in interviewing groups, adults, and
children and ideally, speak the local language. The facilitator needs to have strong
negotiation skills to ensure that the final results and areas for improvement are mutually
agreed upon and shared. It is important that the service provider being appraised understands
that the assessment is carried out in the spirit of true partnership to better determine the areas
where support should be concentrated. Otherwise, the assessment may yield to
disengagement and lack of interest in future cooperation among partners.
4.6. Excel database
Currently, all data is collected using an Excel spreadsheet. Most country teams have enough
knowledge to enter the data in Excel; however, more advanced technical skills are required to
fully manipulate and analyse the data, skills that are largely absent across the organisation.
Additionally, over the years as the data gets collected, Excel will not provide enough
resources to track progress over time. Save the Children is looking into developing a database
to collect, host and analyse data on its global indicators which should be operational by 2014.
5. WAY FORWARD IN MEASURING IMPACT
Save the Children and other like-minded organizations need to be able to measure the impact
their programmes are have in the lives of the beneficiaries. We learned from our experience
that one of the most effective ways to demonstrate impact is by tracking changes in the lives
20
of the children targeted by our programmes over time. While the data collected to track such
changes can be powerful and illustrative, the methodologies used to collect and manage such
data are quite demanding on the capacity of the interviewer. The person collecting such data
needs to be skilled in techniques to interview children, must have a strong understanding of
child safeguarding issues and be able to protect children’s safety, as well as emotional and
physical wellbeing before, during and after the interviews. Additionally, such data
management methods require the interviewer to observe and analyse the family and social
environment of the child, which may not only be ethically challenging, but also costly and
time-consuming. The reality is that in many of the countries where we work, professionals
with such a high level of capacity as well as resources to support this level of data collection
at scale are few. It would be unrealistic to develop an effort-intensive indicator for all country
programmes with a hope that it will be adequately managed. Hence, Save the Children has
decided that changes on the lives of children will be monitored only in the contexts where
country offices have adequate capacity and resources; therefore, using data collection
methods that require individual assessments of children in their community and family
environments is not suitable to be rolled out globally. Save the Children hopes to use the
Quality of Services indicator as a proxy indicator to monitor outcomes achieved, with the
assumption that the services meeting the minimum standards have a positive impact in the
lives of children. We understand that this assumption has several limitations and to be able to
make judgements on the extent of the impact achieved, the findings drawn need to be
triangulated with other sources, such as evaluations, reviews, case studies and other sources
of information.
After piloting the Quality of Services indicator in seven countries, Save the Children has
decided to roll it out globally, which means that all country offices with relevant programmes
must report annually on this indicator. After two phases of piloting, certain adjustments and
21
improvements were made to the methodology, tool and the guidelines. For example, a new
section on how to develop improvement plans and how to use the data for decision making
and provision of targeted technical support has been added to the guidelines to help country
offices to focus their support on the areas where improvements have been mutually identified
with the service provider. This was done to ensure that the assessment score and the progress
planned are mutually owned. The initial version of the tool did not have the red/amber/green
categories, and, as a result of the assessment, the service providers either passed or failed the
quality standard. This proved to be discouraging for the service providers and did not allow
country offices to track and demonstrate progress adequately. And finally, certain key
definitions were revised to incorporate contextual differences.
The country offices participating in the pilot indicated that not only the results but also the
process of the assessment has been has helped to establish an open dialogue with the service
providers. The service providers found the tool and the assessment process an effective
learning opportunity: some of them expressed interest in modifying the methodology and
using it as a self-assessment in the coming years. They mentioned that the standards and the
indicators clarified for them what mechanisms they should have in place to increase the
likelihood that their services are making a difference in the lives of children.
The experience of rolling out global indicators has shown that it takes up to two years in Save
the Children to receive meaningful data from 80 country offices. Therefore, once rolled out,
during 2013 and 2014 the organization will focus on building capacities on reporting against
this indicator. It is likely that only the data for year 2015 (reported in February 2016) will be
publically available. During this period, any interested organisations are welcome to use the
guidelines and tools available thus far, and their recommendations and insights on the use of
these would be appreciated.
22
6. ANNEXES: Standards and indicators per service provider
6.1. Formal Service Providers
Standard Indicators Data source
1: Information
about the
service
1.1 Information about the service, it’s purpose and services available is provided to
children and adults through an appropriate method of communication (e.g. in local
languages, through written and verbal communication)
- Service provider
- Children and adult service users
- Observation
2: Child
safeguarding
2.1 A child safeguarding policy and/or Code of Conduct is in place for service staff and
users
- Service provider
- Children and adult service users
- Observation
- Review of policy document
2.2 Service provider has a mechanism in place for receiving and responding to
complaints from children
- Service provider
- Children and adult service users
- Review of complaints process documentation
2.3 All staff have been checked before hiring for suitability to work with children and in
line with any national regulatory requirements
- Service provider
- Review of job descriptions and safeguarding
check documents
2.4 Service provider maintains an appropriate adult/child ratio (this well depend on type
of service and any national regulatory requirements)
- Service provider
- Observation
23
3: Compliance
with national
regulations
3.1 Service provider has acquired relevant government licensing or accreditation
- Service provider
-Review of licensing or accreditation document
4: Equality and
non-
discrimination
4.1 Children using the service do not experience discrimination because of differences
arising out of gender, age, disability, religion, ethnicity, cultural background, language or
other personal factor
- Service provider
- Children and adult service users
- Observation
5: Child
participation
5.1 Children have an opportunity to contribute their views to the running of the service - Service provider
- Children and adult service users
5.2 Children’s wishes are sought and taken into account in any decisions which directly
affect them (for example care placement decisions)
- Service provider
- Children and adult service users
6: Staff
experience,
training and
support
6.1 Staff have any specialized training relevant for their role and appropriate national
qualifications where they exist
- Service provider
6.2 Staff receive on-going training, supervision and support which is specific to their role - Service provider
7: Placement
planning and
review
7.1 Efforts are made to facilitate contact with and return children to a community based
placement if safe and appropriate (e.g. family care, kinship care, foster care, custodial
care or adoption)
- Service provider
- Children and adult service users
7.2 Where necessary every child (or family) has an individual care plan recording
significant needs and issues
- Service provider
- Children and adult service users
8: Privacy 8.1 An appropriately screened private place is available for interviews or meetings with
children
- Service provider
24
- Children and adult service users
- Observation
9: Case
management
9.1 Staff have received training on case management - Service provider
9.2 Service takes appropriate steps in identifying, reporting, planning, referral and
follow-up of cases
- Service provider
- Children and adult service users
9.3 Service has good links with other service providers (e.g. health, psychosocial,
education, legal, welfare) and is part of active system for referral of children and families
who need support
- Service provider
10: Records
10.1 An accurate record/case note is made for every child (or family) using the service - Service provider
- Observation
10.2 All records/case notes are securely stored - Service provider
- Observation
10.3 Staff maintain the confidentiality of children’s personal and case information - Service provider
- Children and adult service users
- Observation
11:
Effectiveness
11.1 Service demonstrates that it prevents abuse and violence by supporting vulnerable
children and families, and identifying issues early
- Service provider
- Children and adult service users
11.2 When a concern is raised about a child, service provider establishes their immediate
well-being and takes any necessary protective action
- Service provider
- Children and adult service users
25
11.3 Children and families receive timely support (do not have to wait long periods) - Service provider
- Children and adult service users
- Observation
11.4 Service is accessible to children and families in terms of hours, location and fees - Service provider
- Children and adult service users
- Observation
6.2. Community Based (Informal) Providers
Standard Indicators Data source
1: Information
about the
community
mechanism
1.1 Information about the community based provider, it’s purpose and support available is provided to children
and adults through an appropriate method of communication (e.g. in local languages, through written and verbal
communication)
- Service provider
- Children and adult
service users
- Observation
1.2 Community based (informal) provider holds public meetings to inform and involve the community at large
of their activities related to child protection
- Service provider
- Children and adult
service users
- Observation
2: Roles and 2.1 Community-based (informal) provider has developed a terms of reference - Service provider
26
responsibilities
2.2 Community-based (informal) provider has developed an action plan or activity plan - Service provider
2.3 Community-based (informal) provider holds regular member meetings - Service provider
2.4 Community-based (informal) provider has established regular linkages with and coordinates with relevant
local government officials at town and sub-district level (e.g. social workers, district child protection
committees, police, education and health workers)
- Service provider
- Observation
3: Community
ownership and
representation
3.1 Membership of the community (informal) provider is diverse and incorporates representatives from various
groups within the community (men, women, religion, disability status, young, elderly, unemployed, etc)
- Service provider
- Children and adult
service users
- Observation
3.2 Appointments to membership are made through non-discriminatory and transparent selection process - Service provider
- Children and adult
service users
4: Child
safeguarding
Community (informal) provider has a child safeguarding policy and/or Code of Conduct in place - Service provider
- Observation
4.2 A mechanism exists for receiving and responding to complaints that children know how to access - Service provider
- Children and adult
service users
5: Equality and
non-
discrimination
5.1 Children using the community (informal) provider do not experience discrimination because of differences
arising out of gender, age, disability, religion, ethnicity, cultural background, language or other personal factor
- Service provider
- Children and adult
service users
27
- Observation
6: Child
participation
6.1 Children have an opportunity to contribute their views to the running of the community based (informal)
provider either through direct involvement or indirect participation (e.g. through links with children and youth
clubs)
- Service provider
- Children and adult
service users
6.2 Children’s wishes are sought and taken into account in any decisions which directly affect them - Service provider
- Children and adult
service users
7: Staff
experience,
training and
support
7.1 Members of community- based (informal) provider have any specialized training relevant for their role and
appropriate national qualifications where they exist
- Service provider
7.2 Members of community- based (informal) provider receive supervision from the government or civil society
organization (NGO or INGO) taking on the supervisory role at least every 3 months
- Service provider
7.3 Community- based (informal) provider report on activities at least every 3 months to the government or civil
society organization (NGO or INGO)
- Service provider
- Observation
8: Referral and
Case
management
8.1 Members of community- based (informal) provider have received training on case management - Service provider
8.2 Community based (informal) provider take appropriate steps in identifying and referring children at risk - Service provider
- Children and adult
service users
- Observation
8.3 Community- based (informal) provider has good linkages and coordinates with relevant government - Service provider
28
authorities and formal service providers who handle child protection cases
9: Records 9.1 Community based (informal) provider maintain the confidentiality of children’s personal and case
information
- Service provider
- Children and adult
service users
- Observation
10:
Effectiveness
10.1 Community based (informal) provider demonstrate a good understanding of child protection - Service provider
- Observation
10.2 Community based (informal) provider demonstrate that they prevent abuse and violence by supporting
vulnerable children and families, and identifying issues early
- Service provider
- Children and adult
service users
- Observation
10.2 When a concern is raised about a child, the community based (informal) provider establishes their
immediate well-being and takes any necessary protective action
- Service provider
- Children and adult
service users
- Observation
