Kiwis Count 2.0 Technical Report Version 2.1 (July 2015)

This technical report was written for the State Services Commission by Nielsen.

Contents:

1.0 Introduction 3

2.0 Background 3

3.0 Questionnaire 4

3.1 Questionnaire development 4

3.2 Questionnaire review process 6

3.3 A modular approach 6

4.0 Sampling 8

4.1 Sampling frame 8

4.2 Target population 8

4.3 Sampling process 9

4.4 Sampling for first iteration of Kiwis Count 2012 10

5.0 Continuous Surveying and Fieldwork 11

5.1 Initial contact – invitation letter 11

5.2 Second contact – first reminder postcard 12

5.3 Third contact – survey pack 12

5.4 Final contact – second reminder postcard 13

5.5 Exceptions – February and December mailout process 14

6.0 Data Entry / Editing and Cleaning 15

7.0 Response to Survey 16

7.1 Response rate 16

7.2 Mode of response 17

7.3 Cut off dates 18

8.0 Sample Composition 20

9.0 Weighting 21

10.0 Data Analysis Explanation 22

10.1 Overall Service Quality Scores Calculation 22

10.2 2014 Changes to Kiwis Count calculations 23

10.3 The Rolling Average Calculation 23

10.4 The Statistical Significance Calculation 23

10.5 Suppression Rules 24

Appendix 1 Questionnaires 25

1

Appendix 2 Service Groupings 40

Appendix 3 Questionnaire Change Log 42

Appendix 4 Data Entry Protocols 53

The Kiwis Count survey and this report contain material from Citizens First, used under licence and reproduced with the permission of the Executive Director of the Institute for Citizen Centred Service. This work is licensed under the Creative Commons Attribution-Noncommercial-No Derivative Works 3.0 New Zealand licence. [In essence, you are free to copy and distribute the work (including in other media and formats) for non-commercial purposes, as long as you attribute the work to the Crown, do not adapt the work and abide by the other licence terms.] To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/3.0/nz/. Attribution to the Crown should be in written form and not by reproduction of any such emblem, logo or Coat of Arms.

2

1.0 Introduction

The purpose of this document is to outline the technical details of the Kiwis Count Survey, including methodology, sampling, weighting and data analysis.

Kiwis Count is a continuous survey which allows the approach to the survey to be improved from month to month. This document was updated after more than three full years of surveying. The examples in this report are based on recent examples where appropriate and reflect new reporting procedures which have been implemented for June Year End 2014 reporting onwards.

The State Services Commission (SSC) publishes survey findings in separate reports on a regular basis.

2.0 Background

The Kiwis Count survey is an all-of-government national survey measuring New Zealanders’ experiences of public services. Kiwis Count uses the Canadian “Citizens First” survey methodology. Canada is among the world leaders in providing citizen-centred public services. By adapting international best practice for use in New Zealand, this allows useful comparisons to be made between New Zealand’s public service and those of an acknowledged world leader.

The Kiwis Count survey was first carried out by the State Services Commission in 2007 (www.ssc.govt.nz/kiwis-count-research-survey) and then repeated in 2009 (www.ssc.govt.nz/kiwis-count-2009 ). The purpose of the 2009 survey was to measure New Zealanders’ experiences of public services, comparing against the 2007 baseline to measure progress and identify opportunities to further improve frontline service delivery.

Both the 2007 and 2009 surveys were point-in-time measures. Each was conducted via a postal survey (with an option to complete online). While providing valuable information, the outputs were periodic data releases and written reports that provided static performance measurement.

In 2012, Kiwis Count changed to become a continuous survey. This change allowed the capture of trend information and provides regular updates so the survey is a business-as-usual management tool rather than an occasional one-off project reviewing performance. Every month, a number of New Zealanders are invited to participate in the survey. Data is provided quarterly to SSC via SPSS and Excel tables.

Each quarter SSC publishes an update outlining results, in particular any significant movements and trends.

3

3.0 Questionnaire

The questionnaire (included in Appendix 1) reflects the most recent questionnaire (as at Feb 2015) and is based on the questions from the 2007 and 2009 surveys. (Copies of all past questionnaires are available at http://www.ssc.govt.nz/kiwis-count.)

The core of the Kiwis Count survey asks New Zealanders about the services they have used in the past 12 months and the quality of these services. It asks respondents about their experience of 42 government services (a list of services included in the survey can be found in Appendix 2).

Respondents are also asked in more detail about their most recent experience. Specific questions are asked depending on the type of service experience (telephone, online, or face-to-face).

Perceptions of public services versus the private sector are also evaluated, with the quality of service received from private sector organisations (such as banks, insurance companies and Internet service providers) measured for benchmarking purposes.

The survey is also used by other government agencies as a cost effective way of providing other information. For example, the Feb 2015 survey includes questions about

· the “pain points” of dealing with government services for the Department of Internal Affairs’ Result 10 team

· privacy and security of personal information provided to the government for the Office of the Government’s Chief Privacy Officer.

A full list of the changes to the questionnaire is included at Appendix 3.

The survey concludes with an interchangeable module (see below section 3.3 A modular approach) and demographic questions.

3.1 Questionnaire development

Prior to the establishment of the project, SSC determined which parts of the questionnaire to review based on their experiences from the 2007 and 2009 surveys. Where possible, question wording remained consistent with previous Kiwis Count surveys to allow comparisons.

However, the sequence of questions was amended to make navigation through questions clearer and more logical for respondents. The questionnaire was divided into four sections as follows:

Section A: Experiences of public services (the key service quality section - virtually unchanged from previous measures)

Section B: Experiences of non-government services

Section C: Contact with public services

Section D: Demographic section

These changes were tested during the pre-testing process. Pre-testing was carried out during November 2011, involving 10 interviews with a cross-section of the population as follows:

· Two males and eight females

· Nine of European ethnicity and one of Pacific ethnicity

· Two aged under 25 years, four aged 25 to 39 years and four aged over 40 years

· A mix of employment status including employed, student, beneficiary and retired

· Those who have had recent experiences with the services that were changed between 2009 and 2012 (see table 3.1.1 below)

4

In particular, the pre-testing reviewed changes to the services listed and to other question wording changes, as well as general questionnaire order and flow. It also reviewed reaction to, and persuasiveness of, the communication materials.

The details of, and reasons behind, the questionnaire changes related to the list of services respondents are asked to consider at the start of the questionnaire, are outlined in the table below. A full list of all questionnaire changes can be found in Appendix 3 of this report.

TABLE 3.1.1: CHANGES TO THE LIST OF SERVICES

2009 wording 2012 wording Reason for change

A drivers licence, registering a vehicle or changing ownership of a vehicle

Obtained, renewed, changed or replaced a driver licence

Licensed or registered a vehicle

In 2009 this service had very high usage. However, there were several aspects of a service rolled into one rating.

This service has been split into two, each now rated separately (following consultation with NZTA).

Sickness, domestic purposes or unemployment benefit

Receiving a benefit such as the sickness, domestic purposes or unemployment benefit

Clarification for respondents

State or council rental accommodation

Living in a Housing New Zealand home

The question was more useful to Housing NZ if council rental accommodation was excluded.

New service added in 2012

ERO (Education Review Office) school or early childhood reports

ERO provided information showing many New Zealanders used these services and provided feedback on the question wording

New service added in 2012

Visited sorted.org.nz for information to help manage your personal finances or retirement income’

Previous research by the SSC on the drivers of satisfaction with online services showed that many New Zealanders use these services

A visa or permit to work in New Zealand

Removed in 2012 Low usage in the 2009 survey

Citizenship Removed in 2012 Low usage in the 2009 survey

Funding assistance for a business or a business grant

Removed in 2012 Low usage in the 2009 survey

2012 wording 2014/2015 wording Reason for change

A kindergarten that your child attends or may

A kindergarten, day-care, crèche, preschool, home-based service, playcentre,

Wider range of early childhood services added to reflect all available services (following

5

attend in the future Kōhanga Reo, Aoga Amata, Puna Reo or playgroup etc that your child attends or may attend in the future

consultation with the Ministry of Education)

Receiving a benefit such as the sickness, domestic purposes or unemployment benefit

Receiving a benefit such as Jobseeker Support, Sole Parent Support or a Supported Living Payment

Updated to reflect new benefit categories as at 15 July 2013 (following consultation with the Ministry of Social Development)

Visited sorted.org.nz for information to help manage your personal finances or retirement income

Visited sorted.org.nz for information to help manage your personal finances or planning for retirement

Change to wording to be more inclusive of all retirement options (not just those related to retirement income) (following consultation with the Office of the Retirement Commissioner)

A university, polytechnic or wananga about a course you are attending or may attend in the future

A university, polytechnic or wānanga about a course you are attending or may attend in the future

Accent added to the word wānanga

3.2 Questionnaire review process

The full questionnaire is generally reviewed every 6 months, to ensure it continues to meet its objectives. Feedback from respondents, received via cognitive testing and via comments made by those who have completed the survey, are also monitored to ensure the questionnaire remains easy to understand and to complete.

3.3 A modular approach

Whilst the majority of questions remain unchanged each quarter to continue the time series, some questions have been changed to gather new information and there is a section of the questionnaire (Section C) that takes a modular approach enabling completely new information gathering. This section can be changed as frequently as quarterly, dependant on information requirements and the printing of hard-copy materials. The table below outlines the major changes to the questionnaire and modules employed so far:

[Note Appendix 3 is a log of all changes to the Questionnaire.]

TABLE 3.3.1: QUESTIONNAIRE CHANGES AND MODULE SCHEDULE

Date Module name

February - December 2012 Contact with public services

January 2013 -

Government and the digital environment (note questions amended at January 2014 and then augmented at June 2014) Also question at A4 was amended at January 2014 to enable relative satisfaction of single or multi-channel use information

6

to be gained

January 2014 - June 2014 Ministry of Education questions

June 2014 - New question added at A10 to gather information on the “pain points” of dealing with public services

January 2015

New question added at A13 to gather information on the privacy of personal information New question added at A5 and A6 to gather information on perceptions of staff effort involved in providing service

The questions about contact with public services from the 2009 survey were retained in February to December 2012 to allow comparisons to be made.

7

4.0 Sampling

4.1 Sampling frame

The target population for this research is New Zealanders aged 18 years and over. The sampling frame for the Kiwis Count Survey is the New Zealand Electoral Roll. This was also the case for the 2007 and 2009 surveys.

The Electoral Roll allows a random selection of named respondents as well as allowing more targeted oversampling of those of Māori ethnicity (identified as being of Māori descent in the Roll). It also allows more targeted sampling by age group, as it provides age data (within five-year bands).

To ensure the sampling frame is up-to-date with most recent addresses, an updated version of the Electoral Roll is used every three months.

To ensure that people selected at random and invited to take part in the survey are not re-contacted more than once within a 12-month period, steps are put in place when an update of the Electoral Roll is used. As there is no unique identifier on the Electoral Roll, a special process is undertaken. This includes a search involving combinations of names, addresses, occupations and meshblock to track respondents who have already been contacted. It is impossible to track all respondents based on this search, however a hit rate of 88% was achieved when identifying participants selected in February 2015 (the most recent Electoral Roll update).

4.2 Target population

The target population for the survey is based on population estimates from Statistics New Zealand. The table below shows the target population proportions for 2012 and for the latest 2013 census data. The 2013 census data has been used for the processing of the July-September 2014 quarter onwards. Prior to that the 2012 estimates were used.

8

TABLE 4.2.1: POPULATION ESTIMATES

Age

2013 Census Population

%

June 2012 Population Estimates1

% 18-24 years 13 14 25-34 years 16 17 35-44 years 18 18 45-54 years 19 19 55-64 years 15 15 65 years or older 19 18 Gender Male 48 48 Female 52 52 Ethnicity NZ European 71 69 Māori 11 11 Pacific 5 5 Asian 11 9 Other 8 12 Region Auckland 33 34 Upper North Island 22 20 Lower North Island 24 22 South Island 20 24 1 Ethnicity population data is based on 2006 Census data

Note that respondents are able to record multiple ethnicities, hence this breakdown adds to more than 100%. 4.3 Sampling process

Over the course of a year a minimum of 2000 surveys are completed (500 per quarter). The sample is selected on a monthly basis (see section 5.0. Continuous Surveying and Fieldwork). Key aspects of how the sample database is selected each month are as follows:

· The sample is drawn in iterations in the following order:

1. A random sample of Māori electors (based upon the Māori descent indicator field in the full Electoral Roll). Māori electors are oversampled, at a rate which is adjusted as required, to ensure sufficient completed surveys are received, based on response rates from this group in previous months

2. A random sample of Youth (aged 18 to 24) is selected using the age band field in the Electoral Roll. Youth are also oversampled at a rate that is adjusted as required

3. The remaining sample is a stratified sample (by area) of all non-Māori descent electors

· To ensure the sample is geographically representative of the New Zealand population, sample is drawn from each TLA in proportion to population using the 2013 census data from Statistics New Zealand.

9

The minimum number of surveys completed each year, for each target group, is outlined below.

TABLE 4.3.1: MINIMUM NUMBER OF COMPLETED SURVEYS

Target Group Incidence in population

Minimum number of completed surveys each

year

Māori 15% 300

18 – 24 13% 260

All other - 1440

TOTAL 2000

The number of invitations sent per target group is flexible and is constantly adjusted based on the responses achieved in previous months.

4.4 Sampling for First Iteration of Kiwis Count 2012

In February 2012, the first month of fieldwork, the number of survey invitations sent was double that of a typical month. This acted as a ‘pilot’ stage for the new methodology and also ensured the first release of data in August 2012 would include a full six-month sample size of approximately 1,000 (February – June 2012). This oversampling in February has been repeated each year since 2012 (repeated for 2013, 2014 and 2015), given the typical lower response to surveys in general over the summer holiday period.

Initially, oversampling was also undertaken to help ensure that the achieved sample proportionately represented Pacific peoples. However, targeted sampling from the Electoral Roll for Pacific people is inefficient compared with Maori or youth oversampling, as unlike these two groups Pacific people are not identifiable from the details contained in the Roll. It required oversampling people living in the 1000 meshblocks across New Zealand known to contain the highest proportions of Pacific people (thus, most of those over-sampled were still of non-Pacific ethnicity). The rate of oversampling differed over the months it was used (February – June 2012). However this was discontinued from July 2012, the decision being that the negative impact on the response rate of over-sampling Pacific meshblocks out-weighed the benefit of a slightly increased number of Pacific respondents.

10

5.0 Continuous Surveying and Fieldwork

Fieldwork is continuous, with a new sample of 432 chosen from the Electoral Roll and sent invitations to complete the survey each month. The following diagram illustrates the timing of communications to potential respondents each month. (Note international addresses are not sent to).

FIGURE 5.0.1 TIMING OF COMMUNICATIONS TO POTENTIAL RESPONDENTS

Respondents complete the survey in their own time, which means there are completed surveys received most days of the month. More details about each communication are provided below (section 5.1 through to 5.4).

5.1. Initial contact – Invitation letter

An invitation letter, which contains a link to the online survey and provides an individual login ID and password, is sent to all those selected from the Electoral Roll to take part in the survey. As of the start of 2015 the letter also includes a QR (quick response) code which enables a respondent to access the survey quickly using a mobile device such as a cell phone or tablet. The letter gives no indication that the respondent will be given the opportunity to complete the survey on paper at a later date. This approach is taken to maximise the proportion of respondents who complete the survey online, thus reducing the amount of data entry required as well as printing and postage costs.

The letter directs respondents to an 0800 number if they have any questions about the survey. Those without internet connection are able to contact Nielsen on the 0800 number and are advised at this stage that they will be receiving a paper copy survey by post in the near future.

The invitation letter is sent on the first Thursday of every month to the full sample of potential respondents.

11

5.2 Second contact – first reminder postcard

The invitation letter is followed up with a reminder postcard, also with the survey link and individual login ID and password.

The reminder postcard is sent the Monday following the initial invitation letter. The reminder postcard is also sent to the full sample of potential respondents. Again, no reference is made to a paper copy being posted.

5.3 Third contact – Survey Pack

Approximately one week later, those who have not yet completed the survey online are sent a survey pack with a cover letter, hard copy questionnaire, reply paid envelope and a Kiwis Count pen. The survey link and individual login ID and password are repeated in the letter should the respondent prefer to complete online.

12

The survey pack is sent the Tuesday of the week following the reminder postcard being sent (i.e. approximately one week later).

5.4 Final contact – second reminder postcard

Those who have still not replied approximately one week later receive a final reminder postcard:

The final reminder postcard is sent the Wednesday following the survey pack (i.e. approximately one week later).

Each month, this communication cycle is repeated, although data is collated and reported quarterly as illustrated below.

FIGURE 5.4.1 OUTLINE OF A TYPICAL QUARTER

Data into SPSS and Excel

13

A 0800 number ( staffed by Nielsen) is provided on each of the communication pieces. This number is available for respondents to call to ask questions about the survey and / or request paper copies of the questionnaire.

5.5 Exceptions - February and December mailout process

In order to avoid the possible negative impact of a lower response rate by conducting fieldwork close to Christmas and the holiday period, a different process is followed for the February and December mailout.

No field work takes place in January, instead a double sample is used for the February mailout i.e. invitation letters are sent to 864 people rather than 432. In December, the invitation letter is sent out one week after the November mailout, with subsequent correspondence following the normal dates as shown in Figure 5.0.1 above. The February mailout process has been in place since 2012. The December mailout process was put in place in December 2013 following a low response rate in December 2012.

14

6.0 Data Entry / Editing and Cleaning

As completed questionnaires are returned to Nielsen’s Wellington office, they are data entered directly into Confirmit, the same software programme used for the online component of the survey. Using the same software removes the chance of error in combining data sources.

The data enterer has different access to the survey tool from a survey respondent. For example, the data enterer has the ability to select ‘no response’ for any question where a hard copy respondent has not selected a response.

Data entry protocols from 2009 have been adopted again for consistency purposes. The 2009 protocols and their adaption to the 2012 survey and beyond are included in Appendix 4.

As part of Nielsen’s quality control processes, 10% of data-entered surveys are verified.

15

7.0 Response to Survey

7.1 Response Rate

To calculate response rate, every individual sent an invitation to complete the survey is tracked and the outcome of the invitation recorded.

A call-log tracks which of the letters, postcards or questionnaire packs are returned as ‘gone, no address’, as well as any telephone notification of refusal to participate. This log also records notifications from third parties that the nominated respondent is not available or capable to complete the survey due to age, language issues, health reasons, death or other disabilities. Every effort is made to remove any respondent from subsequent samples.

If a respondent is having difficulty completing the survey they are able to call the 0800 number (staffed by Nielsen) and ask for assistance. Alternatively, the respondent can ask for someone to complete the survey on their behalf. However the respondent must be present and it must be their experiences or opinions that are recorded and not those of the person helping them complete the questionnaire.

The response rate is calculated as follows:

Completed surveys / total number of invitations mailed out (excluding GNAs and ineligibles) x 100

Ineligibles are defined as those who are unable to participate due to age, language issues, health or other disabilities.

The table below is an example of response rate calculations using data from the October-December 2014 quarter of interviewing.

TABLE 7.1.1 RESPONSE RATE CALCULATION EXAMPLE OCT-DEC 2014

Total surveys mailed out (a) 2592

Ineligibles (b) 159

Gone no address 105

Unable to participate (age, language, health / disability) 54

Completes (c) 1209

Online 652

Hardcopy 557

Incomplete eligibles (d) 1292

Refused (0800 number) 15

Did not hear back from 1261

Survey not fully completed 16

Response rate c/(a-b) 50% The response rate for Kiwis Count is released on a quarterly basis in the quarterly updates and is based on two quarters of data.

The State Services Commission has a strong focus on increasing this response rate over the life of the survey. The continuous approach to the survey means that changes can be made to the methodology each month and the impact of these changes on the response rate can be monitored.

16

Below is a table of some of the initiatives that have been trialled to evaluate their impact on response rate.

TABLE 7.1.2 RESPONSE RATE IMPROVEMENT INITIATIVES

Date Response rate initiatives

Apr 2012 Oversampling rate decreased, thank you line added to postcard

Jun 2012 End date included on cover letter, final postcard prompts for survey to be returned the following week

Jul 2012 Pacific sample group removed

Sep 2012 Change password to survey code

Nov 2012 FAQ document introduced

Apr 2013 New pen included in survey pack (some old stock still used)

Feb 2014 New letter layout, new (longer) questionnaire, new pen included in survey pack (some old stock still used)

Mar 2014 Change to timing of survey materials

Feb 2015 QR code added and FAQ document updated

7.2 Mode of response

On average (based on response rate results since 2012), 56% of respondents are choosing to complete the survey online, with the remaining 44% returning hard copy questionnaires.

For all completed surveys the method of completion (whether online or hardcopy) is captured in the survey tool. This allows for the proportion of completed online and hardcopy questionnaires to be calculated.

Using the Oct-Dec 2014 response rate data as an example (see table 7.1.1 above), the proportion of online and hardcopy completes for that period is calculated as follows:

Online proportion = Completed surveys (n= 1209) / number of online completes (n=652) = 54%

Hardcopy proportion = Completed surveys (n= 1209) / number of hardcopy completes (n=557) = 46%

17

The pattern of response is illustrated in the following chart, based on Q4 2014 data.

FIGURE 7.2.1 RESPONSES FOR OCT-DEC 2014

7.3 Cut off dates for inclusion of data

There is limited control over when a respondent will complete this survey. For example, a respondent that is sent an invitation in February (Q1) for a number of reasons may not complete the questionnaire until May (Q2). Because reporting is carried out on a quarterly basis, protocols were agreed for cut off dates for inclusion in a particular quarter’s data and for reporting of response rate.

The approach that is consistently applied is that a completed questionnaire will be recognised in the month in which it was completed online or received in the post (not the month it was sent out). In the above example, this survey will be included in the Q2 data (for processing).

For simplicity, the same protocol is applied for reporting of response rate: that is, any survey that is received after the cut off date for the quarter will be counted in the following quarter’s response rate calculation.

18

The final dates for inclusion are shown in the table below:

TABLE 7.3.1 CUT OFF DATES FOR QUARTERLY RESPONSE RATE

Month questionnaire completed: Quarter:

Cut off date for inclusion in quarter’s

data and response rate:

July Quarter 1 (financial year reporting)

(referred to as Q3 for processing) 3rd Sunday October August

September

October Quarter 2 (financial year reporting)

(referred to as Q4 for processing) 3rd Sunday January November

December

January (conducted in Feb) Quarter 3 – Please note there is no fieldwork in January, but February has a

double sample

(referred to as Q1 for processing)

3rd Sunday April February

March

April Quarter 4 (financial year reporting)

(referred to as Q2 for processing) 3rd Sunday July May

June

Imposing a cut off date allows the results to be reported quickly; however, it does mean that late respondents are missed from the report. The first reported response rate is potentially slightly lower than a typical quarter because no late respondents have been carried over. As the survey progresses, the impact of respondents carried over from the previous quarter will be cancelled out by late respondents in the current quarter.

19

8.0 Sample Composition

As an example, the sample composition of the first quarter of surveying (February to March 2012) is outlined in the table below, along with the results from the 2009 and 2007 surveys.

FIGURE 8.0 UNWEIGHTED SAMPLE COMPOSITION

Age

2013 Census

Population %

Q4 2014 (Oct-Dec)

profile n=573

%

June 2012 Population Estimates1

%

Q1 2012 (Feb-Mar)

profile n=537

%

2009 profile n=3354

%

2007 profile n=3653

%

18-24 years 13 8 14 15 17 8 25-34 years 16 9 17 12 15 12 35-44 years 18 14 18 18 20 20 45-54 years 19 22 19 20 19 21 55-64 years 15 20 15 17 15 17 65 years or older 19 27 18 18 14 21 Gender Male 48 45 48 42 45 42 Female 52 56 52 58 55 58 Ethnicity NZ European 71 77 69 68 73 76 Māori 11 6 11 21 12 8 Pacific 5 3 5 8 6 3 Asian 11 9 9 8 7 8 Other 8 5 12 6 7 8 Region Auckland 33 30 34 38 26 25 Upper North Island 22 22 20 20 26 25 Lower North Island 24 24 22 21 23 23 South Island 20 23 24 21 25 27 1 Ethnicity population data is based on 2006 Census data

20

9.0 Weighting

To account for factors such as sample design and non-response bias, the data is weighted each quarter before reporting. The purpose of weighting is to adjust the sample to represent the overall New Zealand population.

The variables used for weighting are:

· Age

· Gender

· Location (TA)

· Ethnicity.

Weighting is based on the proportions in the adult population of New Zealand (18+) in the Statistics New Zealand 2013 Census results for age, gender, location and ethnicity.

Iterative Proportional Fitting is used to adjust the cell based Age Gender and Area weights by Ethnicity groups. This approach is broadly consistent with that used in the previous surveys.

The composition of the weighting matrixes is determined by the sample obtained for age and ethnicity groupings (within each area) for each survey quarter.

As an example, the weighting distribution for the first quarter of surveying (February to March 2012) is shown below.

FIGURE 9.0.1 WEIGHTING DISTRIBUTION FOR Q4 2014 (OCT-DEC)

21

10.0 Data Analysis Explanation

The Kiwis Count surveys ask New Zealanders to rate services or express opinions using a scale from 1 to 5. This is consistent with Canada’s Citizens First 4 survey, on which the Kiwis Count surveys are based. In the reports from Citizens First 4 & 5, the responses from the five point scales are converted to a scale ranging from 0 to 100. The average of these converted scores is referred to as the service quality score. This is to allow comparison with the results of previous Citizens First surveys. To enable comparisons between Kiwis Count and Citizens First to be made, we have adopted the Canadian approach of converting the five point rating scales to service quality scores. The two scales correspond as follows:

Very poor Very good

1 2 3 4 5

0 25 50 75 100

Where the survey measures satisfaction, a five point scale is used where one equals very dissatisfied and five equals very satisfied. In the report when we refer to the percentage satisfied this equals ratings of four or five, the percentage neutral equals rating of three, and the percentage dissatisfied equals ratings of one or two.

Very dissatisfied Very satisfied

1 2 3 4 5

Dissatisfied Neutral Satisfied

The same approach is taken where people are asked their level of agreement with statements on various attributes of public services, where one equals strongly disagree and five equals strongly agree. Total agree equals ratings of four or five, total neutral equals rating of three, total disagree equals ratings of one or two.

Strongly disagree Strongly agree

1 2 3 4 5

Disagree Neutral Agree

10.1 Overall Service Quality Scores Calculation

The Overall Service Quality Score is calculated by rescaling the result from each respondent’s five point scale (1,2,3,4,5) to a 101 point scale (0,25,50,75,100), as shown above, then calculating an average of these scores from all the services used.

The overall average uses all service experiences, so a respondent who has used ten services contributes ten observations to the overall score and a respondent who has used one service contributes one observation to the overall score.

This same approach is used for service groups (e.g. Health, Local Government) and individual services. The groupings used in 2012 are shown in Appendix 2.

22

10.2 2014 Changes to Kiwis Count calculations

An opportunity arose mid-2014 to review the calculations and reporting conventions for the Kiwis Count survey. This resulted in a change to how the rolling averages were calculated (i.e. the averaging of results across six months) and how statistical significant differences were reported.

All results since 2012 have been recalculated in line with these changes and this has resulted in some minor changes to the value for some services (there are now some additional services which are identified as having significant changes than with the previous significance calculation).

Below is a full explanation of the 2014 Kiwis Count Calculation Changes as detailed in the June 2014 annual report which can be found at the following link: http://www.ssc.govt.nz/sites/all/files/kiwis-count-june2014-annual-report.pdf.

10.3 The Rolling Average Calculation

In order to ensure a sufficient sample size, Kiwis Count quarterly releases report on the past two quarters of data. For example, a September quarterly release, will report on data collected from April to September, and compare these results to those collected between January and June. Using two quarters of data boosts the sample size to over 1,000 per quarterly release.

In the past, calculations for all quarterly results were based on averaging the aggregate results from two quarters of data, i.e.

For example, if the SQS score for a service in the June quarter was 70 and for the September quarter was 72, then the SQS score for the September rolling-average time period would be 71.

Going forward, calculations for all results will be based on pooling two quarters of data into one six monthly sample, i.e.

The two methods will give the same result if the same number of people answered a particular question in both Quarter (n-1) and Quarter (n).

However, if there are more responses in Quarter (n-1) than Quarter (n), averaging the quarters under the old method meant that the responses in Quarter (n-1) were worth less with respect to the final result than the responses in Quarter (n). Therefore, the new method has the advantage that each survey response will be worth the same with respect to the final result of the period. The new method also has the advantage of being less IT intensive to calculate.

10.4 The Statistical Significance Calculation

In the past, calculations for statistical significance in changes in service quality scores were made:

· for quarterly results, using the rolling average samples, and

· for both annual and quarterly results, at the 90% significance level (to be consistent with the methodology used to measure significant change between 2007 and 2009 in the 2009 Kiwis Count report).

Result Period (n) = Result Quarter (n-1) + Result Quarter (n)2

Result Period (n) = Result (Quarter (n-1) sample + Quarter (n) sample)

23

Quarterly Rolling Averages

This meant the statistical significance test for differences in quarterly results was being applied on non-independent samples. This is illustrated below, where if significance is calculated on R2 against R1, sample Q2 is repeated:

Including Q2 in both R1 and R2 had two opposing effects on the significance testing. First, it increased the sample size making the change appear more significant. Second, it averaged out the difference between the quarterly results, making the change appear less significant. The second effect is larger, meaning that the previous method was calculating significance around the 97%-98% significance level.

Going forward, for results from 2012 onward:

· The calculations for statistical significance of changes in quarterly results will remove the common quarter sample from both rolling samples. In this example, calculating the significance of R2 against R1, will mean Q2 is removed from both rolling samples and effectively the significance of Q3 is calculated against Q1.

This is the most commonly used method statisticians use to deal with non-independent samples.

· Calculate significance at the 95% level.

Because the previous method (non-independent samples and testing at 90%) produced results at the 97%-98% significance level, the new method (calculating quarterly significance on independent samples at the 95% level) means more services are identified as having quarterly significant changes than previously published.

10.5 Suppression Rules

Kiwis Count results for questions with unweighted sample counts of less than 50 (i.e. those answered by less than 50 people) should be used with caution. The margin of error around a result, due to sampling, increases as the sample size decreases. Results for questions with unweighted sample counts of less than 25 are suppressed.

Month Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec

Quarter Q1 Q2 Q3 Q4

Reporting R1

R2

R3

where R1 = (Q1 + Q2)/2 Significance testing for R2 cf. R1 = independent t-test Q3 vs Q1where R2 = (Q2 + Q3)/2 Q2 is excluded because it is constantwhere R3 = (Q3 + Q4)/2

24

Appendix 1: Questionnaires

Full copies of previous questionnaires can be found at the links below:

· 2007: http://www.ssc.govt.nz/sites/all/files/kiwis-count-survey-questionnaire2007.PDF

· 2009 (includes Contact Methods (Channels) Module): http://www.ssc.govt.nz/sites/all/files/kiwis-count-survey-questionnaire-2009.PDF

· 2012 (includes second Contact Methods (Channels) Module): http://www.ssc.govt.nz/sites/all/files/kiwis-count-survey-questionnaire2012.PDF

· 2013 (includes Government and the digital environment Module): http://www.ssc.govt.nz/sites/all/files/kiwis-count-questionnaire-2013.PDF

· 2014 January to June (includes Education in New Zealand Module and Government and the digital environment Module): http://www.ssc.govt.nz/sites/all/files/kiwis-count-questionnaire-2014v1-jan-jun14.PDF

· 2014 July onwards (includes new questions in Section C: Government and the digital environment Module and new questions about experiencing public services at A10): http://www.ssc.govt.nz/sites/all/files/kiwis-count-questionnaire-2014v2-jul-dec14.PDF

· 2015 (includes new questions about the privacy of personal information and the inclusions described in the July 2014 onwards Questionnaire): http://www.ssc.govt.nz/sites/all/files/kiwis-count-questionnaire-2015.PDF and as shown below (as at February 2015).

25

Appendix 1: January 2015 questionnaire version

26

27

28

29

30

31

32

33

34

35

36

37

38

Appendix 2 Service Groupings

Passports and citizenship · A passport · Registering a birth, death, marriage or civil union

39

Education and training · A state or state integrated (public) school that your child attends or may attend in the future · A university, polytechnic or wānanga about a course you are attending or may attend in the

future Employment or retraining opportunities · Applying for or receiving a student loan or student allowance · A kindergarten, day-care, crèche, preschool, home-based service, playcentre, Kōhanga Reo,

Aoga Amata, Puna Reo or playgroup etc that your child attends or may attend in the future ERO (Education Review Office) school or early childhood reports

Health · Received outpatient services from a public hospital (includes A & E) · Stayed in a public hospital · Obtaining family services or counselling · Used an 0800 number for health information

Local government · Visited a public library · Your local council about rubbish or recycling (excluding the actual collection of rubbish and

recycling from your household each week) · Your local council about property rates · Your local council about road maintenance · Your local council about a building permit

Environment and recreation · Visited a National Park · A hunting or fishing license · National environmental issues or the Resources Management Act

Social assistance and housing · The Community Services card · Accident compensation for injuries · Sickness, domestic purposes or unemployment benefit · A housing subsidy or accommodation supplement · A childcare subsidy · Living in a Housing New Zealand home · A rental property bond lodgement, refund or transfer · New Zealand Superannuation

Border services · The arrival process after landing at a New Zealand international airport from Australia · The arrival process after landing at a New Zealand international airport from anywhere

except Australia · Importing goods into New Zealand or customs duties

Justice and security · The Police (for a non-emergency situation) · Paying fines or getting information about fines · Emergency services i.e.111 · A court, about a case you were involved with

Motor vehicles · Obtain, renewed, change or replace a driver licence · Licensed or registered a vehicle

Taxation and business · Enquired about tax, receiving tax credits (such as Working for Families), Student loan

repayments or KiwiSaver · Contact with Statistics New Zealand for information or about taking part in a survey

40

· Importing goods into New Zealand or customs duties · Registering a new company or filing an annual return for a registered company · Visited sorted.org.nz for information to help manage your personal finances or retirement

income · Registered a business entity for tax purposes or filed a tax return

41

Appendix 3 Questionnaire Change Log

Changes between the 2009 and 2012 surveys

Questions A4-10 (identification of most recent contact method, rating on specific aspects of this most recent contact method and overall satisfaction with most recent contact) replaced 2009 sections ‘Your most recent experience’ Q5 – 8

This section now asks people about their most recent experience. Specific aspects of service are rated depending on the channel used (telephone, online, or face-to-face). The specific aspects rated by channel are those known to be the key drivers of satisfaction with that channel.

These changes were made to ensure all questions asked were relevant and meaningful to respondents.

The ‘drivers of trust’ statements asked in 2009 have been reduced to three specific trust questions: 1 1. Trust based on most recent experience 2. Trust based on overall perception of the public sector (A11) 3. Perceptions of trust in the private sector

The perceptions of trust in the private sector have been added to enable benchmarking against public service results.

Question D7 (previous Q18) was reworded to: ‘Do you have a long-term disability (lasting 6 months or more) that stops you from doing everyday things other people can do?’

To match the question recommended by Statistics NZ and used in the Population Census

1 Please note that monitoring levels of trust and integrity is primarily carried out through the SSC’s Integrity and Conduct survey.

42

Changes from 2012 onwards

Year

Period in survey or date of action

Question numbers -hardcopy (& online)

Questions Action Reason

2012

2012 Feb A4 and C9

(in the Q3 fin. year. 2011/2012 questionnaire)

(Q4,Q23)

You selected <insert response from Q3> as the service you had used or had contact with most recently. Thinking about all the types of contact you have had with this service, which one method of contact was used most recently

If a public service wants to get in touch with you with some routine personal information how would you prefer they do this (assuming they have all your current contact details)?

Changed to multi response

(but instructed respondents to only select one)

This is to allow for these questions to be mapped with historic data which had these questions as multi response

2012 Jan-Dec C1-C13

(Q15-Q27)

· Which of these methods did you use when looking for information about public services in the last 12 months?

· Overall, how satisfied are you with looking for information about public services online?

· Thinking about all of the methods that you've used when looking for information about public services, overall how satisfied are you with looking for information about public services?

· Which method do you prefer to use when looking for information about public services?

· Which of these methods did you use when carrying out transactions or

dealings with public services in the last 12 months? · Overall, how satisfied are you with your transactions or dealings with public

services online? · Thinking about all of the methods that you've used to transact or deal with

public services, overall how satisfied are you with your transactions or dealings with public services?

Removed for Jan 2013

Channels questions

43

· Which method do you prefer to use when carrying out transactions or dealings with public services?

· If a public service wants to get in touch with you with some routine personal

information how would you prefer they do this (assuming they have all your current contact details)?

· Thinking about the cell phone services listed below, which of these have you already done using a cell phone or would you do if the option was available to you?

· Have you had dealings or contact with public services over the Internet, in the

last 12 months? · Which of the following best describes why you did not use any public services

over the Internet in the last 12 months? · What would encourage you to use public services over the Internet (or to use

these more often)? 2013

2013 Feb D1,D2

(Q28,Q29)

Thinking now about your day-to-day internet usage, have you used any of the following services in the last 12 months?

Why haven’t you used the internet at all in the last 12 months?

Removed D1 and D2 were removed from the survey (since they were for the purpose of analysing the old module). Replaced by Digital Q's.

2013 Feb C1,C2,C2a

(Q37,Q38, Q39)

How easy is it for you to complete your transactions with government in a digital environment?

Have you done any of the following in the past 12 months?

Thinking about each of the following, how easy was it for you?

Added New questions added focusing on the ease of dealing with Government in a digital environment (R10 questions)

2014

2014 Feb C1

How easy is it for you to complete your transactions with government in a digital environment?

Amended wording

44

(Q37)

Was: A digital environment includes any transactions done over the internet, on a smart phone, at a kiosk or using a Customs "SmartGate"

Changed to: A digital environment includes any transactions done online, over the internet, on a smart phone, at a kiosk or using a Customs “SmartGate”

2014 Feb A1,A2

(Q1,Q2)

In the last 12 months have you personally used or had contact with a public service organisation about any of the following?

Was: A kindergarten that your child attends or may attend in the future

Changed to: A kindergarten, day-care, crèche, preschool, home-based service, playcentre, Kōhanga Reo, Aoga Amata, Puna Reo or playgroup etc that your child attends or may attend in the future

Was: Visited sorted.org.nz for information to help manage your personal finances or retirement income

Changed to: Visited sorted.org.nz for information to help manage your personal finances or planning for retirement

Amended wording

2014 Feb A4a, A4b

(Q40, Q40b)

Was: You selected <insert response from Q3> as the service you had used or had contact with most recently. Thinking about all the types of contact you have had with this service, which one method of contact was used most recently?

· Visited an office or location · Received a visit · Sent a letter · Received a letter · Sent a fax

Amended wording and additional follow-up question added

45

· Received a fax · Sent an email · Received an email · Called on the telephone · Received a call on the telephone · Visited a website looking for information about public services · Visited a website for transactions or dealings with public services (including:

ordering something, applying for something, booking or paying for something online)

Changed to: You selected <insert response from Q3> as the service you had used or had contact with most recently. Please select all the contact methods you had in the last 12 months that related to <insert response from Q3 >.

· Visited an office or location · Received a visit · Sent a letter · Received a letter · Sent a fax · Received a fax · Sent an email · Received an email · Called on the telephone · Received a call on the telephone · Visited a website looking for information about public services · Visited a website for transactions or dealings with public services (including:

ordering something, applying for something, booking or paying for something online)

Changed to: Which one method of contact was used most recently?

2014 Feb -Jun D1-D4

(Q41-Q44)

How many children do you have, or are a primary carer for, in each of the following groups?

· Preschool-aged children who attend an ECE or early learning service (e.g. Kindergarten, day care, crèche, preschool, home based service, playcentre,

Added Ministry of Education initiative which also included boosting the sample

46

Kōhanga Reo, Aoga Amata, Puna Reo, or playgroup) · Primary school-aged children (school year 0-6) · Intermediate school-aged children (school year 7-8) · Secondary school-aged children (school year 9-13)

You mentioned you have a pre-school aged child who attends an ECE or early learning service. Please rate your satisfaction with your child's ECE or early learning service in each of the following areas:

· The ECE or early learning service overall · How well your child's culture is supported · How well your child's language is supported · How well your child is included in activities run by the service · The information you receive about your child's wellbeing (e.g. physical,

emotional) · The information you receive about your child's learning · The information you receive about how you can support your child's learning at

home · The opening hours of the service

You mentioned you have a primary school-aged child or intermediate school-aged child (school years 0-8). Please rate your satisfaction with your child's school in each of the following areas:

· The school overall · How well your child's culture is supported · How well your child's language is supported · The information you receive about your child's wellbeing (e.g. physical,

emotional) · The information you receive about your child's learning · The information you receive about how you can support your child's learning at

home · The information you receive about your child's progress · How well your child is included in school activities · How approachable your child's teacher is

47

· How welcome staff make you feel at school

You mentioned you have a secondary school-aged child (school years 9-13).

Please rate your satisfaction with your child's school in each of the following areas:

· The school overall · How well your child's culture is supported · How well your child's language is supported · The information you receive about your child's wellbeing (e.g. physical,

emotional) · The information you receive about your child's learning · The information you receive about your child's progress towards achieving

NCEA (if applicable) · The information you receive about how you can support your child's learning at

home · The advice you receive about how subject choices link to your child's future

study or employment options · How well your child is included in school activities · How well the school is preparing your child for tertiary education · How well the school is preparing your child for work · How well the school is preparing your child for life when they leave school · How welcome staff make you feel at school

2014 Feb -Jun E1-E7

(previously D1-D7)

(Q30-Q36)

· Gender · In which one of the follow age groups do you belong? · What is your personal income from all sources, before tax or anything else

taken out? · What is your highest completed educational qualification? · Do you have a long-term disability (lasting 6 months or more) that stops you

from doing everyday things other people can do? · Which ethnic group do you belong to? · From time to time, the State Services Commission undertakes other research

projects. Would you be willing for us to contact you in the future to see if you are interested in taking part in such research for the State Services?

Moved from section D to section E for Feb – Jun 2014

(Moved back to section D from Jul 2014)

To allow for Education questions

2014 Jul - A10 Thinking now about all the times you have personally used or had contact with a public Added

48

(Q45)

service in New Zealand in the last 12 months, which of the following happened to you?

· A situation where you have had to provide the same information to several different government agencies

· A government agency making you provide too much information to prove who you are

· A government agency appearing not to have kept information you have provided them with in the past

· A situation where you wanted to complete a whole transaction with a government agency online, but being unable to do so

· Finding out that you had missed out on getting something from government because you did not know it was available

· A government agency not seeming to understand the effect their decisions and requests would have on you

· A government agency that seemed to be looking for a reason to turn down your requests, rather than truly considering your request on merit

· A government agency failing to treat you with proper respect and empathy

· Finding out that you could have avoided being penalised or fined if you had better information about what you had to do

· Different government agencies or staff in a single government agency providing you with conflicting information

· A situation where you had to approach several different government agencies before finding the one who could actually deal with your enquiry

· A problem with one government agency which could have been solved if they had communicated with another government agency

· None of these have happened to me 2014 Jul - A11,A12

(previously A10,A11)

(Q10,Q11)

Overall, thinking about all the different kinds of public services provided by government, circle one number for each statement to indicate your level of agreement or disagreement with the statement.

· Public sectors have a more difficult task than the private sector · I find the quality of service provided by public services to be higher than the

private sector · I expect public services to provide a higher level of service quality than the

private sector

Moved To allow for new A10 question as specified above

49

Again it is your overall impressions we are interested in from what you know or have heard from family, friends or the media.

Overall to what extent do you trust the public service?

2014 Jul - C2,C3

(Q46,Q47)

Thinking about the last time you transacted with government in a digital environment, please indicate your level of agreement or disagreement related to this occasion.

· The process was smooth and efficient · I had to contact too many different places to complete the transaction

(By different places we mean methods (such as online as well as telephoning or visiting an office to speak to someone, sending mail or email) or different agencies / departments).

The next time you need to do this transaction, would you do it online (as opposed to visiting an office or over the phone etc)?

· Yes, I would do it online again · No, I would prefer to do it a different way · I won’t be needing to do this transaction again · Don't know / unsure

Added

2014 Jul - C4, C4a (previously C2, C2a)

(Q38, Q39)

For each of the services listed below please select the statement that best describes whether you have used it in the past 12 months.

· Been through New Zealand Customs (online refers to a SmartGate) · Made a booking with the Department of Conservation (such as DoC hut) · Renewed a passport · Applied for an IRD number, filed an individual tax return or paid tax · Applied for financial assistance from Studylink, Work and Income or Senior

Services · Paid a Police fine · Paid for a vehicle licence

Then rate how easy the service was to use.

Moved To allow for new C2, C3 questions as specified above

50

2015

2015 Feb A1,A2

(Q1,Q2)

In the last 12 months have you personally used or had contact with a public service organisation about any of the following?

Was: Receiving a benefit such as the sickness, domestic purposes or unemployment benefit

Changed to: Receiving a benefit such as Jobseeker Support, Sole Parent Support or a Supported Living Payment

Amended wording

Updated to reflect new benefit categories

2015 Feb A5, A6

(Q5, Q6)

Thinking about your most recent service contact, indicate your level of agreement or disagreement related to using this service on this occasion.

· Staff went the extra mile to help you get what you needed

Added Additional statement

2015 Feb A13

(Q48)

Thinking now about the privacy of your personal information when dealing with public services, for each statement indicate your level of agreement or disagreement.

There is no right or wrong answer, it’s just your opinions that we’re interested in.

· I am satisfied that any personal information I provide to government agencies is properly protected

· Government agencies are doing a better job at keeping personal information safe than they were 12 months ago

Added New questions about the privacy of personal information

2015 Feb D3

(Q32)

What is your total annual personal income from all sources, before tax or anything else is taken out?

Amended wording

Reference frame added (i.e. annual)

2015 Feb D4

(Q49)

Which best describes your household’s total annual income before tax?

· $0/none · $1 - $5,000

Added New question for household income

51

· $5,001 - $10,000 · $10,001 - $15,000 · $15,001 - $20,000 · $20,001 - $25,000 · $25,001 - $30,000 · $30,001 - $35,000 · $35,001 - $40,000 · $40,001 - $50,000 · $50,001 - $70,000 · $70,001 - $100,000 · $100,001 - $150,000 · $150,001 - $200,000 · More than $200,000

52

Appendix 4 Data Entry Protocols

The following data entry protocols from 2009 have been adopted again for consistency purposes. The 2009 protocols and their adaption to the 2012 survey are shown in the table below:

TABLE A3.1 2009 DATA ENTRY PROTOCOL ADOPTION FOR THE 2012 SURVEY

2009 Protocol Adaption for 2012 Adoption

Blank responses (which were left blank) No adaption required

Selection of multiple items, where only one should be selected (all codes entered in the comment box and questions subsequently recoded to multiple response)

No adaption required

Providing a service rating for a service, when the service itself had not been identified in Question 2 (service was re-coded as having been selected)

No adaption required

Also applied to B1

The description in Q5B clearly does not match the description of the Q2 item that was entered in Q5a; and it was not possible to identify a single service from Q2 (this was dealt with by coding Q5a as 43 (incorrectly answered) and any answers until Q9A were noted to be skipped/crossed out)

Adopted for question A3

Coded as 97

If no service had been selected for Q5a and more than one service has been selected and rated in Q2. (Coded as 43 “incorrectly answered” at Q5a)

Adopted for question A3

Coded as 97

If the service selected in Q5a had not been selected and rated in Q2 (coded as 43 “incorrectly answered” at Q5a).

Adopted for question A3

Coded as 97

New protocols adopted in 2012:

· Any online or hard copy response is counted as a completed survey so long as it has been completed as far as question A2

· If a respondent has answered the wrong set of questions at A5-A8 (based on their response to A4), their responses are transferred to the correct question where they align. Where there is no alignment they are coded as not answered

· If a hard copy respondent selects more than one service at A3, their response is coded as not answered

· If a hard copy respondent writes “superannuation” as a response to D5, their response is coded as not answered

· If a qualification is not easily aligned to the existing codes or is not legible and the respondent is not able to be contacted for clarification, their response is coded as not answered

To enable these edits to be made, the data enterer has different access to the survey tool from a normal respondent. For example the data enterer has the ability to select ‘no response’ for any question where a hard copy respondent has not selected a response.

53

New protocols adopted in 2014:

· If a hard copy respondent rates a service at C2, when the service itself has either not been selected or selected as not having done this, the service answers will stand.

· At A4, if selected something in column B which has not been selected in column A, the answer at B will be transferred to A also.

· If a respondent has answered the wrong set of questions at D2-D4 (based on their response to D1), their responses are transferred to the correct question where they align. Where there is no alignment they are coded as not answered.

New protocols adopted in 2015:

· If total annual personal income is greater than household total annual income, then ‘not answered’ is selected at household total annual income.

54