Embed Size (px)
Citation preview
1
Proposals for consultation
December 2019
New STAR framework
2
The new STAR framework
Contents The new STAR framework .................................................................................................................................................................... 2
1. About this document ............................................................................................................................................................... 3
2. About STAR ................................................................................................................................................................................... 3
3. Summary of consultation and analysis findings ....................................................................................................... 4
3.1. Social landlords ................................................................................................................................................................. 4
3.2. Residents ................................................................................................................................................................................... 5
3.3. Key driver and survey variable analysis .................................................................................................................... 6
4. The proposed new STAR framework.............................................................................................................................. 8
4.1. The questions ..................................................................................................................................................................... 8
4.2. Core questions .................................................................................................................................................................. 8
4.3. Question order and number.................................................................................................................................... 12
4.4. Collection method ........................................................................................................................................................ 12
4.5. One-off or ongoing ...................................................................................................................................................... 13
4.6. Valid response scales ................................................................................................................................................ 14
4.7. Valid response options .............................................................................................................................................. 15
4.8. Sampling and statistical reliability ........................................................................................................................ 16
4.9. Outsourcing ...................................................................................................................................................................... 16
4.10. Data protection and anonymity ....................................................................................................................... 17
4.11. Sharing and comparing results ........................................................................................................................ 17
5. Summary of new framework against landlord and resident objectives .................................................. 19
5.1. Landlord objectives ..................................................................................................................................................... 19
5.2. Resident objectives ..................................................................................................................................................... 20
6. Thank you and next steps .................................................................................................................................................. 20
3
1. About this document This document summarises the outcomes of the HouseMark-led review of the STAR1 methodology
for collecting and measuring resident feedback. It proposes a new STAR framework for the social
housing sector.
We are now entering the final phase of consultation and are keen to collect views from landlords,
residents, research specialists and other stakeholders before publishing the final framework in
January 2020.
Our primary method of collecting feedback is via an online survey which can be found here.
However, if you would prefer to provide feedback in a different way, please contact
[email protected]. The deadline for consultation responses is 20th December 2019.
The review is led by HouseMark with support from research specialists Acuity and TLF Research.
Oversight is provided by:
• National Housing Federation (NHF)
• Chartered Institute of Housing (CIH) and CIH Cymru
• Association of Retained Council Housing (ARCH)
• National Federation of ALMOs (NFA)
• Councils with ALMOs group (CWAG)
• Tenant Participation Advisory Service (TPAS) and TPAS Cymru
• Tenants and Residents of England (TAROE)
• Community Housing Cymru (CHC)
• Welsh Local Government Association (WLGA)
2. About STAR STAR stands for ‘Survey of Tenants and Residents’ and was originally developed by HouseMark in
2011 as a consistent framework for social landlords to collect and report on resident satisfaction.
Since then STAR has been widely adopted by the sector with over 300 landlords regularly carrying
out STAR surveys.
Carrying out a valid STAR survey is currently a regulatory requirement in Scotland but remains at
present voluntary for social landlords in England, Wales and Northern Ireland.
The main aim of the STAR framework is to provide landlords with consistent best practice in the
collection of resident feedback, and to enable them to compare and share results with tenants and
stakeholders in a meaningful way.
In light of the challenges set out by the social housing green paper for England and recent
technological and best practice innovations in the field of customer feedback, HouseMark recently
carried our wholesale review of the STAR framework. The review aimed to develop a modern yet
consistent framework for measuring customer satisfaction that allows like-for-like comparisons,
focusses on what matters to residents, and enables landlords to make use of the data to drive
improvements.
1 STAR stands for Survey of Tenants and Residents
4
3. Summary of consultation and analysis findings HouseMark, supported by Acuity and TLF Research have carried out extensive research and
consultation as part of this review. This includes:
- An online survey of social landlords with 257 organisations responding representing over 3
million social housing units.
- Six workshops held across England and Wales with over 150 landlords attending. These
workshops were supplemented by telephone interviews with landlords who were unable to
attend.
- An online residents survey completed by almost 8,000 social housing tenants and
leaseholders.
- Six regional workshops for residents hosted by TPAS and attended by tenant representatives
from over 60 social landlords.
- Key drivers and variables analysis using extensive feedback data from both the housing
sector and beyond, held by HouseMark, Acuity and TLF Research.
This section summarises the findings from our research and consultation as part of this review.
3.1. Social landlords Our research found that whilst most social landlords still carry out STAR surveys an increasing
number have reviewed or are reviewing their approach to collecting resident feedback.
- 91% of social landlords carry out a perception survey, but most supplement these with
transactional surveys – most commonly following a repair.
- Whilst the majority of landlords still carry out perception surveys by post or telephone,
increasing numbers are adopting a multi-channel approach and embracing online and SMS
surveys.
- Telephone and postal surveys receive significantly better response rates than digital
methods.
- Perception surveys are twice as likely to be anonymised as transactional surveys.
- Perception survey results are more likely to be reported to boards, tenants and employees
and benchmarked with other organisations.
- Transactional survey results are more likely to be used for staff training, informing service
reviews and linked to staff and contractor rewards.
- Landlords value comparisons with other landlords but many do not believe they are currently
comparing like with like.
- Some existing STAR measures remain very useful to landlords.
- Many landlords are now also asking questions about trust and ease of access / customer
effort.
- The majority of landlords currently struggle to consistently make good use of tenant
satisfaction data to drive service improvement, with some notable exceptions.
- The majority of landlords are reviewing their customer feedback processes with a focus on
automation, digitalisation, and producing more actionable insight.
You can read our landlord survey publication here.
In conclusion, whilst there is no clear consensus amongst landlords on what they would like the new
framework to include, we were able to identify their top five requirements.
5
- Landlords want to listen to residents and need a framework that helps them make better use
of tenant feedback to drive service improvements.
- Landlords are keen to understand how they are perceived generally through perception
surveys that encourage feedback from residents they may not usually hear from.
- Landlords also want to be able to understand customer journeys and quickly identify points
of failure through the use of targeted transactional surveys.
- Landlords value being able to compare in and out-of-sector but want to be sure the
comparisons are fair, and variables are understood.
- Landlords are keen to better use modern data-mining techniques such as sentiment analysis
and predictive analytics to get more out of their data.
3.2. Residents The response to our resident consultation was impressive with almost 8,000 residents from over 20
landlords responding to our online survey, and around 150 tenant representatives from over 60
social landlords at regional workshops hosted by TPAS.
Key findings include:
- One fifth of residents have responded to at least one satisfaction survey on behalf of their
landlord in the last two years.
- These residents were marginally more likely to be satisfied than those who had not
completed a survey.
- 8% of tenants had been given the opportunity to complete a survey and had declined. The
most common reason being they felt it was a waste of time as the landlord would not act on
the results.
- Nearly one in ten residents have seen the landlord’s published satisfaction survey results and
just over half felt it reflected their own experiences.
- 56% of residents do not know what the landlord does with survey results, and 25% did not
know their landlord even undertakes resident surveys.
- Respondents who had seen the results of a landlord survey are on average more satisfied
than those who had not, and over five times more likely to think they had benefitted from
changes made by their landlord as a result of feedback surveys.
- In terms of resident priorities, repairs featured prominently in the comments, but 70% of
respondents described the ease of contacting their landlord as equally if not more important.
Better communication was also a prominent theme in the feedback.
- Around one third of residents said it would be very valuable to be able to compare their
landlord to other landlords. A greater proportion (52%) said it would be very valuable to see
trend information – is their landlord improving or getting worse.
- There was significant interest in satisfaction survey results at the resident workshops hosted
by TPAS, but some cynicism of the results published by their landlords.
- Some resident representatives were not convinced landlords do anything with survey results.
- Some involved residents reported experiences of landlords being unable to explain the
results, which eroded trust.
- Involved residents want a consistent and independently verified framework to build trust in
the figures.
- Tenants want a range of survey methods to suit them and liked the idea of being given a
menu to register how to give their views.
6
More detail on the results of the resident survey can be found here.
It is clear that there is some work to be done to build trust with residents that landlords are using the
survey results to improve and are reporting genuinely on their performance. The new framework
presents an invaluable opportunity to address this.
Our top five resident priorities for the new framework are:
- Residents want their voices heard and want landlords to be transparent around their results
and what they do with the feedback provided.
- Residents want a consistent framework that allows them to genuinely understand how their
landlord is performing and are particularly keen to see how their landlord is improving over
time.
- Different residents prefer to be surveyed in different ways, and would value being able to
express a preference.
- Residents want surveys to be kept short and focussed on what matters to them.
- Residents want to quickly and simply see results in an accessible format.
3.3. Key driver and survey variable analysis One of the biggest barriers when it comes to understanding and comparing resident satisfaction is
recognising and quantifying the impact of survey variables such as demographics, survey
methodology or area.
As part of the review of STAR, HouseMark with support from Acuity and TLF Research have carried
out key drivers and variables analysis using the following sources:
- HouseMark’s extensive database of over 10-years’ satisfaction results and landlord profile
data for over 400 social landlords UK-wide.
- The detailed resident survey data collected out as part of this review, with nearly 8,000
responses
- The detailed landlord survey data collected as part of this review from 257 social landlords
across England and Wales.
- Resident response data collected via TLF Research’s consumer panel.
- Analysis carried out by research specialists Acuity and TLF Research.
- English housing survey data from the MHCLG.
- Helpful research into variables carried out previously, including:
o Rethinking customer insight – HACT 2018
o Housing frontiers – Ipsos MORI 2008
o The state of customer satisfaction in the UK – UKCSI 2019
Whilst a landlord’s performance will of course impact on satisfaction, our research identified two
types of other variable that can also influence results:
- Contextual variables – these are features of the context the landlord operates in and are
largely outside of the landlord’s control (for example location or tenant age).
- Methodological variables – these are features of how you collect resident feedback that can
influence satisfaction results (for example carrying out surveys by telephone compared to
post).
7
Understanding, managing and accounting for variables is key to making sense of high-level
aggregate satisfaction scores. You can read more detail on our variables analysis to-date here.
HouseMark will continue work to quantify variables where we currently have less clarity.
In terms of key drivers for overall resident satisfaction, we found responses to the following
questions types were most likely to correlate with overall satisfaction2:
- Satisfaction with the repairs and maintenance service
- Satisfaction with landlord engagement (this includes ‘softer’ questions such as listening to
views, keeping promises and keeping residents informed)
- Ease of doing business with the landlord.
2 HouseMark’s STAR benchmarking Service – Analysis of findings 2013/14 (April 2015)
8
4. The proposed new STAR framework The remainder of this document details the key features of the proposed new framework.
These are the features proposed by HouseMark and our partners for a new sector-wide
methodology for collecting and reporting on resident feedback.
They have been put together following the extensive consultation and research the project team has
carried out in 2019.
4.1. The questions We propose three tiers of questions:
Core questions – a suite of five perception and two transactional questions to be included in the
appropriate survey and that can be benchmarked with HouseMark.
Recommended questions – a number of additional questions which we recommend for inclusion.
These include the existing STAR questions on VFM for rent and VFM for service charges which
remain reasonably popular with landlords and residents but have been excluded from the core suite
in order to keep the number of top-level questions to a minimum.
Optional questions – an extensive library of optional questions to choose from to build the survey
that is right for your organisation.
The overall themes of the core questions have been agreed, but there remain various options on the
wording of the questions on which we are seeking your feedback.
These options are also currently being tested with residents to understand which questions work
best to provide an answer to each theme.
Detailed below are the core questions under consideration.
Recommended and optional questions will be circulated in due course.
4.2. Core questions Theme Your home
Links to… The Home Standard (England)
Green Paper theme of keeping properties in good repair.
Welsh Standard of Performance guidance note ‘Ensures that all homes
meet required standards of design and quality’.
Potential
questions
To what extent do you agree or disagree with the following statement “I am
happy with my home”.
How satisfied or dissatisfied are you with your home?
How satisfied or dissatisfied are you with the overall quality your home?
(previous STAR question)
Survey type Perception
Rationale Providing a good home is the primary purpose of a social landlord. This
existing question was rated the most important question by residents in our
survey (median importance score 8.3) and third most useful by landlords
(mean usefulness score 8.7).
It also allows for trend analysis with legacy data, which 52% of residents
consider to be very valuable.
9
This question is also a current regulatory question in Scotland.
Notwithstanding, the question remains quite broad and research is
underway to understand what residents are thinking of when they answer
this question.
Theme Health and safety
Links to… The Home Standard (England)
Green Paper theme of maintaining the safety of buildings.
Welsh Standard of Performance guidance note ‘Meets all applicable
statutory requirements providing for the health and safety of the occupants
in the home’.
Potential
questions
To what extent do you agree or disagree with the statement, “My landlord
provides a home that is safe and secure”.
To what extent do you agree or disagree with the statement, “I feel safe and
secure in my home”.
How satisfied or dissatisfied are you that your landlord keeps your home
safe and secure?
Survey type Perception
Tier Core
Rationale Health and safety is currently a top regulatory priority following on from the
Grenfell tragedy. It features as a green paper theme and is highly likely to
feature in a future suite of regulatory metrics in England.
It is important that a question of this nature is a perception question, and not
reliant on an interaction with the landlord.
Theme Your neighbourhood
Links to… The Neighbourhood and Community Standard (England)
Green Paper theme of responsible neighbourhood management.
Welsh Standard of Performance ‘A track record of achieving positive
outcomes, responding appropriately to new challenges and performance
issues’.
Potential
questions
To what extent do you agree or disagree with the statement, “My landlord
makes a positive contribution to my neighbourhood”.
To what extent do you agree or disagree with the statement, “I feel safe and
secure in my neighbourhood”.
How satisfied or dissatisfied are you with your neighbourhood as a place to
live? (previous STAR question)
Survey type Perception
Rationale Measuring a landlord’s contribution to neighbourhoods and communities is
difficult and is best approached via a perception measure.
However, the existing STAR question on neighbourhoods is currently one of
the lowest rated questions by landlords (mean usefulness 7.4), many of
whom feel the actions they can take to improve performance against this
measure are limited.
Residents on the other hand rated this question one of the most important
(mean score 8.0).
We are therefore testing alternative questions, and what residents are
thinking of when they answer them, in order to see if there is a viable
10
alternative that achieves the Green Paper objectives.
Theme Responsive repairs
Links to… The Home Standard
Green paper theme of keeping properties in good repair
Welsh Standard of Performance ‘Effective and appropriate tenant
involvement and high quality and improving services’. Potential
questions
Overall, how satisfied or dissatisfied
are you with the repairs service you
received on this occasion?
Have you had any repairs carried
out in this property in the last 12
months? - If 'yes': Thinking about
the LAST time you had repairs
carried out, how satisfied or
dissatisfied were you with the
repairs service provided by {your
landlord/LANDLORD NAME}
Survey type Transactional Perception
Rationale Repairs is a key driver for overall satisfaction with a landlord. It was also
ranked by residents as a priority (mean importance 8.0).
However, many landlords do not find the current perception question useful,
as it does not allow them to react promptly to issues or effectively manage
their repairs operatives / contractors. 85% of landlords now collect overall
satisfaction with the repairs service via a transactional survey, and higher
figures reported from transactional surveys when compared to perception
surveys have the potential to cause confusion and erode trust. In our
landlord workshops most landlords reported that satisfaction with repairs
lends itself more to a transactional method, allowing them to identify and
rectify issues early.
As such we are proposing that the core repairs satisfaction measure is
collected via transactional surveys, with the option to collect via perception
surveys by asking beforehand if they have had a repair in the last twelve
months (as is the case for the regulatory measure in Scotland). We are
currently testing the impact of this approach with residents.
Theme Customer service / ease of access
Links to… Tenant involvement and empowerment standard (England)
Green paper themes of respectful and helpful engagement with residents
and effective handling of complaints.
Welsh Standard of Performance ‘Effective and appropriate tenant
involvement and high quality and improving services’.
Potential
questions
How easy did you find it to deal with
[your landlord] on this occasion?
Have you contacted your landlord
in the last 12 months? - If 'yes':
Thinking about the LAST time you
contacted your landlord, how easy
did you find them to deal with?
How satisfied were you with the
customer service received on this
occasion?
Have you contacted your landlord
in the last 12 months? - If 'yes':
Thinking about the LAST time you
contacted your landlord, how
11
satisfied or dissatisfied were you
with the customer service
received?
Survey type Transactional Perception
Rationale Customer services/ease/effort came out as a strong theme in our landlord
survey, with 43 landlords asking for it to be added to the new framework.
Residents also scored this highly, with customer service achieving a mean
average score of 8.2 and ease of contacting the landlord 8.0.
Measuring the customer experience lends itself more to transactional
surveys (of any kind that involves dealing directly with the landlord).
Landlords want to be able to review customer journeys and pick up on
problems as they arise. However, a perception alternative should be
provided for landlords who do not do transactional surveys. We are
currently testing the impact of this approach with residents.
Theme Resident involvement and empowerment
Links to… Tenant involvement and empowerment standard (England)
Green paper theme of respectful and helpful engagement with residents
Welsh Standard of Performance ‘Effective and appropriate tenant
involvement and high quality and improving services’.
Potential
questions
To what extent do you agree or disagree with the statement, “My landlord
keeps me engaged in a way that suits me”.
To what extent do you agree or disagree with the statement, “My landlord
listens to residents’ views and takes notice”.
To what extent do you agree or disagree with the statement, “My landlord
listens to residents’ views and acts on them”.
To what extent do you agree or disagree with the statement, “I trust my
landlord”.
To what extent do you agree or disagree with the statement, “My landlord
treats me with respect”.
Survey type Perception
Rationale The existing question ‘Listens to views and acts on them’ is not highly rated
by landlords (mean usefulness 7.9) or residents (mean importance 7.7).
Landlords at our workshops said they felt the question tried to ask two
things at the same time and was therefore difficult to analyse.
However, key driver analysis shows that the existing question is the second
biggest driver of overall satisfaction after repairs, and there is a view from
both residents and landlords that a question that measures resident
involvement and empowerment is important.
There is little consensus however on what the alternative question might be.
Whilst ‘trust’ featured as a common theme amongst landlords, as a question
it was not highly rated by residents (mean importance 7.3 and a view from
the resident workshops that the question was a bit too ‘touchy-feely’).
As such we have collated the five most likely candidates put forward by
both residents, landlords and sector leaders and are opening them up for
consultation. We are also testing all five with residents as part of the final
consultation.
12
Theme Overall
Links to… Consumer regulation agenda in England.
Welsh Standard of Performance guidance note ‘Demonstrates how the
board assures itself of current service performance, including tenant
satisfaction, and drives continuous improvement’.
Potential
questions
Taking everything into account, to what extent do you agree or disagree
with the statement, “I have a good landlord”.
Taking everything into account, how satisfied or dissatisfied are you with the
service provided by [your social housing provider]? (previous STAR
question)
Survey type Perception
Rationale
for
proposed
question
This is the existing question rated most useful by landlords (81% described
as very useful).
It is also the question most likely to be reported to executives, boards and
tenants (83%) and retaining it allows for trend analysis.
Net Promoter Score was considered as an alternative as this measure is
used in many sectors. However, NPS proved divisive for landlords (mean
usefulness score 6.5 – the lowest of the existing measures) and was not
popular with residents (mean importance score 7.1 – the lowest of the
potential measures tested).
The alternative put forward by the sector is currently being tested with
residents.
4.3. Question order and number We are proposing that landlords retain the flexibility to add as many questions to the core suite as
they would like, in any order. These questions may be taken from the HouseMark library of
recommended and optional questions, or alternatively can be different questions developed by the
landlord that align with their strategic objectives.
We did not find any evidence of the order of the questions having a significant impact on the results.
However, we would make the following recommendations:
- Surveys should be kept as short as possible to avoid survey fatigue.
- Questions should be worded correctly, or so similar as to make no difference, and no
preamble should be added.
- In perception surveys, landlords should include open-ended questions to give them a better
understanding of the responses provided and allow for sentiment analysis. For example:
‘What could we do better?’.
- Landlords should also include a two-part question to see if respondents are willing to have
their name attributed to the feedback (see later section on data protection and anonymity).
Landlords will be able to benchmark any or all or the core questions with HouseMark and/or Acuity.
4.4. Collection method 45% of landlords now take a multi-channel approach to collecting responses (i.e. they use two or
more collection methods). This is a significant increase on 2015 when only 5% of landlords took a
multi-channel approach.
13
In part this is down to increased digitalisation of the resident base making online surveys more viable,
but landlords who have adopted a multi-channel approach also advise it has helped boost return
rates and access more views from different resident segments.
Residents at our workshops also expressed a preference for being given a range of options for how
they would like to be surveyed.
Notwithstanding, our research shows that different collection methods can produce differing
response rates as per the table below. These are mean averages based on HouseMark’s landlord survey,
but in practice response rates can vary significantly.
Collection method Face-to-
face
Telephone Post SMS Online
Average response rate 90% 50% 37% 22% 16%
Our research has also identified a negative survey bias with online surveys when compared to
telephone surveys, and a positive survey bias for postal surveys. You can read more on this on our
analysis of key variables.
Notwithstanding, the new framework proposes to retain and indeed encourage a multi-channel
approach to ensure feedback is received from all segments. Landlords should however have a clear
understanding on the impact of collection method on their results, and how they will combine those
results into aggregate figures.
For benchmarking purposes, HouseMark will apply a weighting that mitigates for collection method
survey bias (see later section on sharing and comparing your results).
4.5. One-off or ongoing An increasing number of landlords are carrying out their perception surveys on a rolling / ongoing
basis. From our survey of 257 landlords carried out earlier in 2019, we found that 35% now carry out
perception surveys on an ongoing basis. This typically involves surveying a random sample of
tenants each week, month or quarter, with a view to building up an annual figure over time.
This is a significant increase on 2015 when fewer than 1 in 10 landlords carried out perception
surveys in this way.
Carrying out surveys of a rolling basis can enable landlords to pick up and act on issues sooner but
makes no significant difference to the overall results reported.
It is proposed therefore that the new framework continues to allow both one-off and on-going
surveys. We embrace the valuable trend towards more surveys being carried out on a rolling basis
but recognise that for some landlords a one-off survey is a more practical and financially viable
option.
Some evidence exists to suggest the season of the survey can have a minor impact on the result,
and this may affect when landlords choose to carry out one-off surveys. We are currently carrying
out more research into this but note that for benchmarking purposes survey results should not be
more than two years old (or three years if under 1,000 units).
14
4.6. Valid response scales 75% of landlords currently use a five-point response scale for their perception surveys. A further
14% currently use a ten-point response scale. Both five and ten-point response scales are
compatible with the current STAR methodology.
The remaining 11% of landlords use a mixture of two, three, four and seven-point scales, with the
most common alternative being four-point. These response scales are not compatible with the
current STAR methodology, and landlords that use them are currently unable to benchmark their
results with HouseMark. Proponents of the four-point response scale view it as better as it prevents
respondents from ‘sitting on the fence’ by choosing a middle or neutral option.
Regardless of what is currently used, when asked whether or not they thought a neutral option (such
as ‘neither satisfied nor dissatisfied’) should be allowed:
• 51% of landlords said yes, there should be a neutral option.
• 36% said that a neutral option should not be allowed.
• 13% said they did not know.
As part of this review, HouseMark has carried out detailed research into the difference using a four or
five-point scale can make. Forcing respondents ‘off the fence’ results in both higher satisfaction and
dissatisfaction, as illustrated in the below table. Thanks to additional out-of-sector data from TLF
Research, we were able to corroborate our findings with other sectors.
Sector Social
housing
Banking Mobile
phone
provider
Water
provider
Hairdres
ser
Postal
service
NHS
Median % satisfied (5-
point)
70.8 79.4 79.2 73.3 81.2 78.9 72.1
Median % satisfied (4-
point)
79.2 88.8 88.5 87.8 90.2 87.8 83.7
Median % dissatisfied (5-
point)
17.6 9.8 8.7 8.5 6.1 8.3 10.9
Median % dissatisfied (4-
point)
25.5 11.2 11.5 12.2 9.8 12.2 16.3
On average across all seven sectors a four-point scale produces a percentage satisfied 10.2%
higher than a five-point scale, and percentage dissatisfied 4.1% higher.
This means that whilst the merits of allowing or disallowing a neutral option can be debated, it is clear
that changing the response scale from five to four points has a significant impact on the results.
Residents however expressed a clear preference for being allowed a neutral option. Taken from our
survey of almost 8,000 residents from a range of landlords:
• 76% preferred a scale that allowed them a neutral option
• 12% preferred a scale that did not allow a neutral option
• 12% said they did not know or were not bothered
These findings were corroborated at our resident workshops where attendees said they felt it was
important to allow respondents a neutral option.
Although in different sectors different response scales exist, most commonly a neutral option is
allowed. The MHCLG annual English household survey uses a five-point scale, as does the current
15
Scottish regulatory framework. The Market Research Society Code of Conduct advises that best
practice should include a neutral option.
In light of both landlord and resident feedback, we have therefore concluded that there is not a
sufficient case to change the existing response scales. The framework proposes therefore to
continue to only allow a five or ten-point response scale. This has the additional benefit of enabling
trend analysis (all other things considered).
No significant difference has been identified between using a five or ten-point scale, although some
analysts value the extra granularity of responses provided by a ten-point scale.
4.7. Valid response options The incumbent STAR framework allows both verbal and numeric response options. Specifically:
• 5-point numeric – A scale of 1-5
• 10-point numeric – A scale of 1-10
• 5-point verbal – E.g. Very dissatisfied; Fairly dissatisfied; Neither satisfied nor dissatisfied;
Fairly satisfied; Very satisfied. As part of our research we are investigating an alternative
verbal scale (Totally disagree; Disagree; Neither agree nor disagree; Agree; Totally agree).
No significant difference in results has been identified through using a verbal or numeric scale.
However, outside of the sector research specialists are increasingly using emojis (smiley faces) or
star ratings as valid response options. These can be particularly useful when carrying out short
online or SMS surveys (but are clearly not applicable for telephone interviews).
According to our landlord survey, 8 landlords (out of the 257 that took part) currently use emojis or
star ratings to collect transactional satisfaction with repairs, and this number is likely to increase.
The introduction of visual response options is still relatively new and research on the impact of the
options is emerging but there does not appear to be any authoritative position.
SurveyMonkey tested the use of several visual rating options and concluded the completion rates
and results we unaffected. A study in the Netherlands (Troepel et al, 2019) analysed the results from
an extensive survey which had employed a number of different response options (smileys, stars,
hearts, coloured and neutral radio buttons amongst others) and compared these to a control group.
The study found that there were some differences, if only slight. Smileys tended to give scores in line
with the traditional radio button response, but stars and hearts gave a slightly lower figure. There is
some evidence that the use of emojis can boost completion rates without damaging data quality on
online surveys; these tended to work best on shorter surveys, and not on sensitive questions which
could lead to respondent uncertainty or discomfort.
In response to the limited impact of using visual rating options when compared to verbal or numeric,
we are proposing that the new framework is extended to allow visual rating options for postal, SMS
and online surveys (as long as they are on a five-point scale). However, where visual rating options
are used, organisations should label the scale to ensure the respondent understands the rating scale
and selects the right option. Organisations should also be careful when using emojis to check the
appropriate context and meaning of each to ensure there is no ambiguity or misunderstanding over
the meaning of the image.
16
4.8. Sampling and statistical reliability When consulted, landlords were generally comfortable with the sampling requirements of the
existing STAR framework for perception surveys, which requires a random and representative
sample of all tenants.
There were no significant objections to the existing statistical reliability requirements, currently set to
a standard confidence level of 95% for all landlords but allowing a more generous confidence
interval for smaller landlords for practical reasons. We are therefore not proposing any changes to
the below table for the new STAR framework:
Population Required confidence
level
Required margin of
error
Minimum number of
replies
Under 1,000 95% +/-5% Up to 275
1,000 – 10,000 95% +/-4% 375 - 565
Over 10,000 95% +/-3% 950 or more
Importantly, the new framework now also contains two transactional survey questions. The same
confidence level and intervals will apply for benchmarking transactional survey data with HouseMark.
Landlords who carry out rolling perception surveys and transactional surveys are likely to report
figures regularly internally that do not meet these sampling criteria, which is perfectly acceptable.
However, landlords should be clear about the sample size in their internal reports and when reporting
figures to residents.
Landlords should also ensure that their sample is representative and use weighting where
appropriate. Landlords carrying out their surveys in house should pay particularly close attention to
representativeness and weighting, which is hugely important for quality research.
You can view the full existing guidance on sampling, statistical reliability and weighting here.
4.9. Outsourcing Our extensive landlord survey revealed that 46% of landlords currently outsource the collection of
perception survey data to a research specialist. Local authorities, ALMOs and smaller housing
associations are slightly more likely to outsource than larger housing associations.
31% of landlords do all their customer feedback collection in house. The remaining 23% of landlords
have a mix of in-house and outsourced responsibilities.
In theory at least carrying out your survey in-house as opposed to outsourcing it should have no
impact on the results. Notwithstanding, there was a view expressed by involved residents at our
TPAS forums that they trust the data more if the collection of feedback is outsourced. Outsourcing
surveys also helps guarantee anonymity in the eyes of residents.
HouseMark data for over 300 landlords who have provided valid STAR satisfaction results for
benchmarking shows a mean average overall satisfaction that is 2% higher for surveys carried out in-
house. This difference is small and well within the margin of error when accounting for other
variables.
As such the new framework will continue to allow for both in-house and outsourced feedback
collection. However, for those carrying out their survey in-house it is recommended that an
17
independent audit is carried out from time to time to help build and maintain resident trust in the
results.
4.10. Data protection and anonymity Data protection has become increasingly important since the last review of the STAR framework in
2015, and landlords should consider the Data Protection Act 2018 when carrying out surveys.
Additional guidance on data protection in relation to carrying out a STAR survey will be provided
when the final framework is released.
According to our landlord survey, 60% of current perception surveys are anonymised, compared to
30% of current transactional repairs surveys. Feedback from our resident workshops suggested that
residents valued the ability to be able to provide anonymous feedback, and there was a perception
that feedback would be more genuine if respondents were allowed to remain anonymous.
For the purposes of this document, we are proposing that STAR compliant surveys are confidential
by default. This means that when carrying out the analysis in-house, only the minimum number of
people in the organisation will be able to identify who has said what as they are involved in the
analysis or are a senior stakeholder of the project. Participants should be informed that this is the
case at the start of the interview. Once the appropriate analysis has been carried out there is no
need for the feedback to continue to have personal identifiers where participants have chosen to
remain anonymous. When using an agency for data collection, feedback can be supplied back to the
landlord anonymised.
However, we recognise that landlords are also keen to be able to identify and rectify issues raised as
part of the feedback. As such we are recommending that all participants are given the opportunity to
have their details attached to their feedback. To capture this, we would recommend asking two
questions at the end of the STAR survey3:
1) The information you have provided in the survey is confidential. However, would you be happy
for us to attach your name to your responses so that [LANDLORD] has better information to
help us/them improve services?
2) If yes, would you be happy for [LANDLORD] to contact you to follow up any of the comments
or issues you have raised?
4.11. Sharing and comparing results One of the significant limitations of the previous STAR framework from a landlord’s perspective is
that is has not always been easy to benchmark results with confidence that you are comparing like-
with-like. This is primarily due to the impact of variables mentioned earlier in this document, and into
which we continue to carry out research.
3 Questions should be adjusted accordingly for in-house surveys
18
From a resident’s perspective, this has to some extent eroded trust in the figures, with a significant
minority of residents from our survey advising that the results their landlord reports do not match
their own perception of the service.
Furthermore, some of our involved residents said that their landlord’s inability to explain the results or
indeed how they compare has led to further cynicism of the figures reported.
It is clear the new framework needs to build trust amongst both landlords and residents that the
figures are credible, and that any comparisons take into account the landlord’s operating context.
To this end HouseMark is taking inspiration from other sectors to propose a ‘star rating ()’ for
landlords who carry out a STAR compliant survey.
We are proposing that landlords are given an overall star rating (), alongside contributory star
ratings for:
- Home (based on perception questions around the quality and safety of the home)
- Repairs (based on transactional survey responses around responsive repairs)
- Customer service (based on transactional survey responses about customer service / ease
of access)
An example representation is included in figure 1 below, for illustrative purposes only. Subject to the
outcomes of the consultation, attractive visuals will be commissioned that reflect the new STAR
brand and logo.
HouseMark star ratings will not be based on your rank within a peer group, but rather based on
achieving certain levels of resident satisfaction that reflect the context you operate in and the
methodology that you have used, taking into account all major survey variables.
Our research shows that star ratings are well understood and liked by the public, and this additional
service provided by HouseMark should help landlords and their residents truly understand how they
are performing in an easy to understand and visually appealing way, whilst avoiding the potential
stigma of league tables.
Our research into quantifying the major survey variables in order to set the star rating boundaries is
still underway, but more information will be released once the consultation completes, subject to the
feedback received.
19
Figure 1: An example representation of star ratings for the social housing sector.
5. Summary of new framework against landlord and resident objectives The below tables summarise how we believe the proposed new framework meets the ten key
objectives identified during consultation.
5.1. Landlord objectives Objective identified New framework response
Landlords want to listen to residents
and need a framework that helps
them make better use of tenant
feedback to drive service
improvements.
The proposed framework has been robustly tested
with residents to ensure it meets their
expectations and gets to the heart of what matters
to them. Landlords can be confident they are using
a reputable framework that lays the foundation for
them to improve.
Landlords are keen to understand
how they are perceived generally
through perception surveys that
encourage feedback from residents
they may not usually hear from.
With new perception questions designed
specifically with the resident in mind, combined
with robust sampling guidance and clear
understanding of variables, landlords can be sure
to get greater understanding of how they are
perceived by all resident sections.
Landlords also want to be able to
understand customer journeys and
quickly identify points of failure
through the use of targeted
transactional surveys.
With the addition of two new transactional survey
questions on responsive repairs and customer
service, landlords can get real-time feedback on
the two most-critical customer journeys and
quickly identify and rectify points of failure.
Landlords value being able to
compare in and out-of-sector but
want to be sure the comparisons are
fair, and variables are understood.
Whilst analysis is ongoing, the new framework
brings clear understanding of key variables.
HouseMark’s new star rating will take into account
contextual and methodological variables to give
landlords a clear idea of where they really stand.
Landlords are keen to better use With new questions that get to the heart of what
20
modern data-mining techniques
such as sentiment analysis and
predictive analytics to get more out
of their data.
matters, whilst also retaining the flexibility for
landlords to add questions that matter to them, the
new framework should provide landlords with
invaluable data for analysis. Including open-ended
questions will bring greater understanding of
responses and enable sentiment analysis.
Guidance on data protection and anonymity will
help ensure you remain legally compliant.
5.2. Resident objectives Objective identified New framework response
Residents want their voices heard
and want landlords to be transparent
around their results and what they do
with the feedback provided.
The proposed framework has been robustly tested
with residents to ensure it meets their
expectations and gets to the heart of what matters
to them. The framework itself does not proscribe
how landlords should use the data, but HouseMark
will release best practice guidance once the
framework is finalised.
Residents want a consistent
framework that allows them to
genuinely understand how their
landlord is performing and are
particularly keen to see how their
landlord is improving over time.
The new framework brings a level of consistency
and transparency to the sector. It has retained
some consistency with the previous framework to
allow trend comparisons, whilst also modernising it
to reflect current best practice. Where the landlord
has been allowed flexibility, our analysis will be able
to quantify what (if any) difference this makes.
Different residents prefer to be
surveyed in different ways, and
would value being able to express a
preference.
The new framework allows a range of collection
methods and has been expanded to allow SMS
surveys and to increase the scope of online
surveys.
Residents want surveys to be kept
short and focussed on what matters
to them.
The number of core perception questions has
been reduced from eight to five, based on resident
feedback on what matters to them the most. The
two transactional questions are designed to fit into
a short transactional survey that could be quickly
and simply completed via SMS.
Residents want to quickly and simply
see results in an accessible format.
The new HouseMark star rating is designed with
residents in mind. It follows best practice in other
sectors and takes account of the landlord’s
context – giving residents and easy-to-understand
and reliable barometer of performance.
6. Thank you and next steps This review would not have been possible without:
- The invaluable input from research specialists Acuity and TLF Research
- The support and guidance received from sector representatives on our steering group,
including: NHF, CHC, CIH, CIH Cymru, ARCH, NFA, CWAG, Welsh LGA, TPAS, TPAS Cymru
and TAROE.
21
- The 270 landlords who have participated in the review, either by providing feedback or
surveying their residents on our behalf.
- The 8,000 residents who responded to our resident survey and the 150 involved residents
who attended the TPAS workshops.
This final stage of the review involves:
- Consulting on the proposals contained in this document with landlords, sector
representatives, residents and research specialists. The deadline for responses is 20th
December 2019, and the best way to provide feedback is via our online survey.
- Testing the proposed questions with residents of 20 landlords who have kindly volunteered
to survey their residents on our behalf, as well as with TLF’s consumer panel.
- We will review all feedback and aim to publish the final framework on 21st January 2020.
If you have any queries about the content of this document or anything else related to STAR, please
contact [email protected]
