Analysis of the characteristics of internet respondents to the 2011 Census to inform 2021 Census questionnaire design

Orlaith Fraser & Cal Ghee

Overview

1. The census and modes of response

2. Census Quality Survey

3. Propensity score analysis

I. Census quality survey results

II. 2011 Census results

4. Conclusions

The census and modes of response

Mode effect = Same respondent gives different response when using different modes

X

Y

2011 Census: Paper by default, Internet option- 81% responded by paper, 19% by internet

2021 Census: Internet by default, possible paper option.

Census Quality Survey (CQS)

What is your date of birth?

01 01 1977

Face-to-face CAPI sample survey.Sample of census respondents asked majority of census questions again.

Answers compared to calculate agreement rates.

CQS answers assumed correct as face-to-face likely to be more accurate than self-completion.

Census Quality Survey

• Stratified Sample region, hard to count, mode

• No adult proxy responses• Individuals weighted

age, sex, ethnic group, mode

• Household representative interviewed

5170 matched

households

9,650matched usual

residents5170

matched households

Paper/CQS agreement rate

Internet/CQS agreement rate

Co

mp

are

agre

emen

t ra

tes

Internet agreement rates significantly higher than paper

Paper agreement rates higher than internet

Possible reasons for differences

Age Scanning errors?

Marital status Social desirability bias?

Disability Recall error?

Group most likely to change answer

Religion

Possible reasons for other differences:• Use of radio buttons• Help information• Scrolling distance• Paper format easier to look ahead

Propensity score method

Direct comparison between internet and paper responders difficult because of differences in

respondent characteristics

Distribution of internet responders matched to that of paper responders

Proportion of internet responders

Propensity Score Method

• Propensity score = Propensity towards exposure to a treatment (responding by internet) given a set of observed characteristics

Proportion of paper respondersAdjustment factor for each subgroup =

1. Individual propensity scores derived from logistic regression model

2. Respondents split into ten subgroups based on propensity scores

3. Internet distribution standardised by applying adjustment factors

STEPS

Propensity Score Analysis of CQS data

VARIABLECQS/Census Agreement

Paper %

Internet %

Internet - Paper

Unadjusted Adjusted

Unpaid Care Agree 82.50 80.46 2.04 -0.26

Disability Agree 85.68 91.99 -6.31 -1.72

Workplace address Agree 43.97 49.31 -5.34 -5.91

Address one year ago Agree 25.51 32.17 -6.66 -3.39

Variables included in logistic regression model:Sex, student status, disability, English as a main language (English or Welsh in Wales), good health and whether working

Propensity Score Analysis of CQS data

VARIABLE

Internet - Paper

Unadjusted Adjusted

Unpaid Care -2.04 0.26

Disability 6.31 1.72

Workplace address 5.34 5.91

Address one year ago 6.66 3.39

Variables included in logistic regression model:Sex, student status, disability, English as a main language (English or Welsh in Wales), good health and whether working

Easier to check postcodes online

Limitations

Household responses may not be independentA highly educated young person may respond online for an older

less well educated person

Proxy effectProxy may not have responded in the same way as the

individual they were representing

Chicken and egg dilemmaAny mode effects included in the model may be considered

as actual predictors

Small CQS sample

Can’t restrict sample to one response per household

Propensity Score Analysis of Census data

More robust analysis using 10% microdata household sample of census data

- stratified by output area

Only household reference persons included Mainly household level variables included as model

predictors Direct comparison of paper/internet proportions for

variable categories rather than CQS/Paper and CQS/Internet agreement rates

Propensity Score Analysis of Census data

Variables included in new logistic regression model:

Age of household reference personCountry of birthDeprivation indicators of a householdEthnic groupHousehold languageHousehold reference person social gradeLiving arrangementsNumber of cars and vans in householdRegionSize of householdTenureUrban Rural classification

Results

Variable Category Paper % Internet %Internet - Paper

Unadjusted AdjustedPensionable age indicator Of pensionable age (65+) 29.32 8.95 -20.37 -1.11Sex Male 58.73 66.34 7.61 2.09Disability Day-to-day not limited at all 73.58 86.06 12.48 -1.99Health Very good health 32.19 43.96 11.77 0.86Hours Works 30-49 hrs/wk 63.52 67.32 3.80 1.30Activity Last Week Not working 39.48 19.24 -20.23 -1.08

Main LanguageEnglish as main language (or Welsh in Wales) 93.83 89.60 -4.23 -0.38

Marital status Widowed 12.66 4.09 -8.57 1.32Marital Status Single 24.83 30.02 5.19 -1.19Level of highest qualifications

Level 2: 5 GCSEs (A* - C) or 1 A level or equivalent 12.42 39.78 27.37 -0.57

All internet/paper differences negligible after adjustment – all explained by differences in population characteristics

Conclusions

• Propensity score analysis can be a useful tool for identifying true mode effects

• Most differences in internet / paper responses attributable to characteristics of respondent group

• Useful tool for testing mode effects prior to 2021 census New mode effects: tablet/mobile/desktop... More detailed knowledge of internet/paper

respondent profiles will help to target support for digitally excluded / reluctant internet respondents

Further information

Please contact:

orlaith.fraser@ons.gsi.gov.uk

cal.ghee@ons.gsi.gov.uk

2021 Statistical Design,Census Transformation Programme,Office for National Statistics