1

1

Web in mix-mode surveys in Web in mix-mode surveys in Norway Norway

Bengt Oscar Lagerstrøm

Symposium on General Population Surveys on the web, London November 2011

2

Surveys in Norway on ”general” populations

• 2001 – Mode effects in a mail plus Internet designed Census. Haraldsen et al (2002)

• 2007 – Rent Market Survey 2006, Lagerstrøm (2007)

• 2008 – Rent Market Survey 2007, Lagerstrøm (2008)

• (2011 – Survey on Education 2011, Haraldsen et al (2012))

• 2011 – CPI - Rent prices, Lagerstrøm (2012)

3

Experiments

1. Census 2001. Mode effects. Paper versus two types of web questionnaires

2. The Rent Market Survey (RMS) – From interviewer administrated to self administrated – effect on RR, bias and estimates

3. RMS – data quality and costs

4. RMS and CPI – panel issues

4

Hypothesis

• Is it essential that the respondent have the opportunity to choose in which mode they would like to participate? A common assumption appears to be that offering multiple modes of reporting makes the reporting task easier, which will lead to higher response and better data quality.

• More motivated respondents and better quality of data are the two main pillars in Statistics Norway’s new data collection strategy, and one way to get more motivated respondents would be to offer them to respond in a preferred mode. But, how should we present the different tools? One by one, all at once?....

5

Option of P or W vs. paper/web (RMS 2006)

0,0

0,2

0,4

0,6

0,8

1,0

1,2

1,4

18-29years

30-39years

40-49years

50-59years

60-69years

70 yearsor older

P - Treatment 3 P1 P - Treatment 3 P3

0,0

0,2

0,4

0,6

0,8

1,0

1,2

1,4

18-29years

30-39years

40-49years

50-59years

60-69years

70 yearsor older

H - Treatment 1 P1 H - Treatment 1 P3

H (15,000) - option to respond on paper/web (telephone)

P (1,000) – option to respond on paper (telephone)

W (1,000) – with option to respond on web (telephone)

0,0

0,2

0,4

0,6

0,8

1,0

1,2

1,4

18-29years

30-39years

40-49years

50-59years

60-69years

70 yearsor older

W - Treatment 2 P1 W - Treatment 2 P3

6

Estimated tenants by treatment (RMS 2006)

 H W P T

Tenants, phase 1 62.9 81.6 58.5 62.4

Tenants, phase 2 61.8 68.3 62.0 62.4

7

Response rates by phases (RMS 2007)

 

           

Phase 1 Phase 2 Phase 3

Seq. Opt. Seq. Opt. Seq. Opt.

           

Total

11.6 18.0 24.7 26.4 47.6 48.6

(1 157) (1 799) (2 465) (2 639) (4 759) (4 861)

Web

11.6 4.7 14.8 6.3 15.0 7.2

(1 157) (472) (1 480) (631) (1 497) (718)

Paper

- 13.3 9.9 20.1 10.3 21.2

- (1 327) (985) (2 008) (1 032) (2 121)

Telephone

- - - - 22.3 20.2

- - - - (2 230) (2 022)

8

Bias by age groups (RMS 2007)

-10,0 -5,0 0,0 5,0 10,0

- 24 år

25 - 34 år

35 - 44 år

45 - 54 år

55 - 64 år

65 år -

Opt Seq

-6,0 -4,0 -2,0 0,0 2,0 4,0

- 24 år

25 - 34 år

35 - 44 år

45 - 54 år

55 - 64 år

65 år -

Opt Seq

-4,0 -2,0 0,0 2,0 4,0

- 24 år

25 - 34 år

35 - 44 år

45 - 54 år

55 - 64 år

65 år -

Opt Seq

P1 P2

P3

9

Costs (NOK) (RMS 2007)

  P1 P2 P3 Total

Cseq 57 000 71 700 256 500 385 200

Copt 85 000 64 000 235 900 384 900

10

Lessons learn from Census and RMS

• Paper seems easier to fill in than web questionnaire, but less errors in web questionnaires for less educated and older respondents

• Don’t give the respondents the options to choose, but change mode to ensure a satisfactory RR.

• Young respondents have a higher propensity to respond on web, especially men

• Estimates could be mode sensitive?

• Probably nothing two win in terms of money if you want same RR and bias structure

11

Our puzzle

Motivation

Availability Opportunity

12

New experiments

• Change the wording in the advance letter – from ”please use the URL and password” to ”A interviewer will contact you, but you can still use the URL/password”

• Sending SMS with direct access to the web questionnaire

13

Mode propensity in a panel survey (RMS/CPI 2011)

  Paper Telephone Web Total

Telephone 90 90 30 60

Web 10 10 70 40