1© GfK 2014 | Considerations around switching from telephone to online survey methodology | November 2014 EU WORKSHOP ON RECENT DEVELOPMENT IN BUSINESS

1© GfK 2014 | Considerations around switching from telephone to online survey methodology | November 2014

EU WORKSHOP ON RECENT DEVELOPMENT IN BUSINESS AND CONSUMER SURVEYSBrussels, 13 – 14 November 2014


Considerations around switching from telephone to online survey methodology

Karsten Shaw and Donna Culverwell, GfK UK


• GfK have been conducting the Consumer research in the UK on behalf of the European Commission since 2006.

• Historically, fieldwork has been conducted as a telephone survey amongst all respondents using a RDD (Random Digit Dialling) sampling methodology

• But, we know that there are more and more considerations /issues with undertaking telephone fieldwork in the UK:

• This is a situation which we do not see any improvement on• Taking this on board, we had to look at alternative ways of undertaking the fieldwork in the

UK; minimising any impact to the long term data trend

• Caller ID in the home • Respondents thinking that the interviewer is trying to sell them something

The background

Respondents becoming more savvy to research calls, meaning they are harder to reach:

More and more respondents are now in mobile only households, which creates issues with reaching people via landlines

Ever increasing cost of RDD sampling methodology


The background

How do you change a methodology that has been running for a number of years without creating shifts in the data?

With the plan to move the UK Consumer survey to a 100% online survey from

May 2015 onwards.

Each month phasing out the number of telephone interviews, whilst increasing the number of online interviews

Working closely with the EC we agreed on a mixed-mode telephone/online approach to

the survey in the UK over a 10 month period (July 2014 – April 2015)

3

1 2

4


Sample Sizes

Aug-14 Sep-14 Oct-14 Nov-14 Dec-14 Jan-15 Feb-15 Mar-15 Apr-15 May-150

200

400

600

800

1000

1200

1400

1600

1800

2000

Telephone Online


Changing the methodology of a tracking study needs to be done with great care

Or is there a need to continue

to track the data consistently over time?

The answer to this question in this instance that we need as much

consistency as possible, to provide a seamless track between results from the existing measures

to the measures going forward

Is the first wave of this ‘new’ study to be treated as a brand new survey,

with no element of consistency going backwards?

This has implications for how the new survey is set up, and what is

done at the early stages to ensure a smooth

transition


Extensive experience of migrating studies to create a seamless transition

• GfK has extensive experience in the art of moving studies from offline to online methodologies to negate the possibilities of movements in the data as a result

• GfK conducts expert statistical analysis to manage such changes

• We are aiming to ensure a smooth transition and to minimise any movement in data from before the move to after

• It is important that the key CCB measures are not affected by changes over the next 12 months. This means keeping all other aspects of the survey unchanged


Parallel run process

• The results of the parallel runs are used to identify the impact of design changes. If the results of the two surveys are the same, then we would conclude that there are no effects and the data from the old and new design can be integrated with no requirement for adjustment

• However, if differences in results are observed, then we would conduct a process of calibration to allow us to bring the two datasets into line. That is, to adjust the results to make them consistent and comparable

• Typically this calibration process would involve:

• Once this calibration process is complete, the two data streams can be fully integrated

Identifying any structural differences in the two samples

Adjusting for these differences through a process of calibration weighting. This will usually correct the majority of differences

Developing an adjustment process to align any remaining differences


Integrating considerations

Once we have a mechanism for adjusting differences between waves, we need to decide which of the data tracks we will adjust.

That is, do we:• adjust historic data to match the new

data trend, or• adjust the new data to match the historic data trend

We usually recommend that we adjust history to match the new data trend/structures. However, in this instance we cannot re-adjust the historic data, so it will require constant re-adjustment of results going forward.


Weighting Methodology


Weighting Methodology

• In this instance, we want to treat the telephone sample as the ‘correct’ view of the world

• There are arguments for and against this, but the main reason is consistency

• Therefore the aim of the exercise is to apply additional weighting to the online data, over and above demographic data, to ensure it resembles the telephone sample as far as possible

• So in the first instance a complete parallel run was conducted in July 2014 of telephone and online


We found some substantial differences in the questions between modesProfiles amongst key questions


Demographic Weighting Targets

Applied equivalently to telephone and online samples


Demographic Differences

Demographics not used in weighting scheme


But demographic weighting does not correct the differences

• Even when weighting by employment, education and income, the differences in the CCB questions persist

• We attempted Propensity Modelling – using Logistic Regression to form propensity weights on the basis of the CCB questions and demographics

• However, this generated extreme weights which had a severe impact on the effective sample size


Propensity weighting had a severe impact on the effective sample size

• Propensity weighting is a common way used to adjust for samples collected from different methodologies

• Logistic regression models were run which generated propensity scores of being an online respondent (compared with a telephone respondent)

• The resulting propensity scores were then divided up into quintiles amongst the telephone respondents

• The cut-offs were then used to generate five groups amongst the online respondents and weights were generated so that the weighted groups were also equal in size


Propensity Weighting

20%Least likely

to be online

20%Most likely

to be online

Logistic Regression modelling is applied to generate a likelihood of being online

These scores are applied to the online sample so that five groups are created

These five groups are transformed back to their original size using weighting

20%

20%

20%

20%

20%


Weights Original Demographic Weights Propensity Weights

Efficiency 97% 32%

Minimum weight 0.66 0.21

Maximum weight 1.61 12.9

Effect of propensity weighting

This does remove most of the differences between the questions, but…


So we used a modified weighting scheme

• The final weighting scheme used adjusts for the differences between the telephone and online samples by using questions which exhibit consistent differences between the modes

• Weights are then generated which minimise these differences and applied to the subsequent wave of online data

• These weights are updated every wave based on the additional information provided


The questions were decided on using the main parallel run data in August

These were the questions which exhibited the most substantial differences in some part of the distribution between telephone and online:

How do you think the general economic situation in this country has changed over the LAST 12 MONTHS?

How do you think consumer prices have developed over the last 12 months?

In comparison with the past 12 months, how do you expect consumer prices will develop in the next 12 months?

In view of the general economic situation, do you think now is the right time for people to make major purchases such as furniture or electrical goods?

Over the NEXT 12 MONTHS, how likely will you be to save any money?

Q.3

Q.5

Q.6

Q.8

Q.11


Weights were generated which removed these consistent differences


These weights are then applied to the subsequent wave of data

• The weights which adjusted these differences were then applied to the subsequent wave of data

• The remaining differences were then examined to ensure that the profile of telephone and weighted online data were similar

• This way, we can ensure an objective weighting procedure which takes into account as much information as possible about previous waves and then applies that to the current wave of data

• The extra wave of data is then used as further information to refine the weights for the subsequent wave


Effect of weighting scheme used

This is a more efficient weighting scheme but some efficiency is still lost

* Most recent wave (October)

Weights Original Demographic Weights Weighting Scheme*

Efficiency 97% 74%

Minimum weight 0.66 0.26

Maximum weight 1.61 3.94


Checklist

Insert chart title here (optional)

We will continue to monitor the weights at each wave and ensure they are fit for purpose without becoming too extreme and adversely impacting the effective sample size

With each extra wave, we also check the weighting strategy to ensure it is still optimal

There will also be annual waves of a full telephone survey to check and refine the weighting scheme

This will ensure a smooth transition from telephone to online without any step change in the trend


THANK YOU

Documents

1© GfK 2014 | Considerations around switching from telephone to online survey methodology | November 2014 EU WORKSHOP ON RECENT DEVELOPMENT IN BUSINESS