From face-to-face to web in a cross-national study

From face-to-face to web in a cross-national study

Ineke StoopSCP/CST

www.europeansocialsurvey.org4th International Workshop on Internet Survey Methods,

Statistics Korea,2012

ESSAcademic survey in 30+ European countriesAttitudes, values and beliefsBiennial, start in 20021+ hour, face-to-face, effective sample size ~ 1500Funders• National Science organisations• European Commission, Framework Programmes• European Science Foundation

Management model• Central coordination• National implementation

http://www.europeansocialsurvey.org/



Aims ESS

To monitor and interpret public attitudes and values within Europe and to investigate how they interact with Europe’s changing institutions

• Provide data on beliefs, attitudes and values for scientific and policy making purposes

• Measure attitude change in a changing Europe

To advance and consolidate improved methods of cross-national survey measurement in Europe and beyond

• Collect data according to highest standards

• Generate methodological research

• Develop and disseminate new best practices

• Develop and improve social survey research infrastructure in Europe




Content questionnaires ESS

Every roundSocio-demo-graphic items

• household composition• education• housing• occupation• social structure

Attitudes and behaviour patterns• religious affiliation and identity• ethnic and national identity• political and social trust• party affiliation and voting

behaviour• media consumption• value orientations• social exclusion

Round specificR1 Immigration

Citizenship, involvement and democracy

R2 Family, work and well-beingOpinions on health and

care- seekingEconomic Morality

R3 Personal & Social Well-beingThe Timing of Life

R4 AgeismWelfare attitudes

R5 Family, work and well-being in times of economic recession

Trust in police and courts

R6 Personal & Social Well-beingUnderstanding, evaluation democracy




Access ESS

Round 1-5 availableFully documentedFree accessTables and graphics online, or download dataTraining package for young researchers

Everything availableAll deviations documented




ESS top data user countries (>50) (June 2012) (N=48,973)

Brazil

Korea, Republic of

Iran, Islamic Republic of

Iceland

Cyprus

Philippines

Bulgaria

Latvia

Turkey

Romania

Ukraine

Israel

Estonia

Sweden

Finland

Austria

Switzerland

France

Poland

Spain

Slovenia

Belgium

0 500 1000 1500 2000 2500 3000 3500 4000 4500 5000




Participation R1 R2 R3 R4 R5

AustriaBelgiumBulgariaCroatiaCyprusCzech Denmark

EstoniaFinlandFrance

GermanyGreeceHungaryIcelandIrelandIsraelItalyLatviaLithuaniaLuxembourgNetherlandsNorway

PolandPortugalRomaniaRussiaSlovakiaSloveniaSpainSwedenSwitzerlandTurkeyUK UkraineTotal 22 26 25 31 28




Methods: high quality

Strictly random sampling• No substitution, no proxy

Population: all inhabitants aged 15+High target response rates• 70% target response rate, max. 3% noncontact

Face-to-face fieldwork• Gold standard, high response rates, universally applicable• Increasingly expensive• Rare in some countries

Pilots, pretesting and fieldwork monitoringAnalysis of cross-national differences• Nonresponse bias, measurement error, sampling design effects




Methods: central coordination

Central specifications for fieldworkClose monitoring national fieldwork (in advance and during fieldwork)Sampling designTranslation proceduresGuidelines response enhancementGuidelines interviewer trainingContact forms, response calculationContext, events




Can we compare countries?

Survey design, methodology,

implementation

Attitudes, values, beliefs

Survey results




Harmonisation strategies (Körner and Meyer, 2005)

Harmonised concept

Harmonised concept

Harmonised concept

Measurementprocedure



National survey/Specific concept



Input harmonisation Ex-ante outputharmonisation

Ex-post outputharmonisation




Output harmonisation

Ex post output harmonisation• Use existing sources

Ex ante output harmonisation• Set up new survey (or develop new instrument)

Output harmonisation• Best national quality• Or, national survey tradition• But, what about optimal comparability




Output harmonisation: problems

• Interview mode• Context question• Lay-out questionnaire• Question text• Introductory text• Answer scales• Handling don’t know• Size of boxes• Use of illustrations




Input harmonisation

Input harmonisation• Design new survey• Optimal comparability

Or, are identical methods and instruments really equivalent in different countries?• But, what about optimal quality




From face-to-face to web in the ESS

Mixed mode research and experimentsDifferent survey modes over timeDifferent survey modes across countriesMixed mode within countries: different composition over timeWhich modes:• Face-to-face• Telephone• Web




ESS studies

Martin, Peter (2011) A Good Mix? Mixed Mode Data Collection and Cross-national Surveys. ASK Research & Methods, Vol. 20 (1, 2011) 5–26.Martin, Peter (2011) What makes a good mix? Chances and Challenges of Mixed Mode Data Collection in the European Social Survey. CCSS Working Paper No. 02.Martin, Peter, and Lynn, Peter (2011) The Effects of Mixed Mode Survey Designs on Simple and Complex Analyses. CCSS Working Paper No. 02.Roberts, Caroline (2007) Mixing Modes of Data Collection in Surveys: A Methodological Review. ESRC National Centre for Research Methods. NCRM Methods Review Papers, NCRM/008.




From face-to-face to web in the ESS

CoverageSamplingNonresponseQuestionnaire designRole interviewersMeasurementAnalysisCross-national comparison




Coverage




Internet access and broadband connections in households, EU27 (%)

2006 2006.5 2007 2007.5 2008 2008.5 2009 2009.5 2010 2010.5 20110

10

20

30

40

50

60

70

80

Internet access Broadband connections




Individuals who used the internet at least once a week, 2011 (%)

IS SE NO NL DK LU FIUK BE DE AT FR EE SK IE

EU27 LV HU MT SICZ ES LT PL CY IT PT GR BG RO

0

10

20

30

40

50

60

70

80

90

100




Individuals who used the internet on average at least once a week, by sex, age and education, EU27, 2011 (%)

All individuals

Men Women 16-24 years

25-54 years

55-74 years

Low education

Medium education

High education

0

20

40

60

80

100




Sampling principles

Identical population definitionComplete coverage• No upper age limits• (Institutional population)• (Non-native language speakers)

Probability samplingEffective sample sizeOptimal sampling design in each country• Address, household, individual• Address and household: selection of target respondent in the field




ESS R2 sampling frames (Häder and Lynn)Country Frame RemarksAustria Telephone book Additional non-telephone households sampled in the fieldBelgium National registerCzech Republic Address register UIR-ADR Select streets, followed by field enumerationDenmark Danish Central Person RegisterEstonia Population registerFinland Population registerFrance None Area-based samplingGermany Registers from local residents’ registration

officeGreece None Area-based samplingHungary Central registryIceland National registerIreland National Electoral RegisterLuxembourg Social Security RegisterNetherlands Postal address listNorway Population registerPoland National register of CitizensPortugal None Area-based samplingSlovakia Central Register of CitizensSlovenia Central Register of PopulationSpain Continuous CensusSweden Register of populationSwitzerland Telephone registerTurkey Cluster of addressesUK Postal address listUkraine None Area-based sampling

Simple random sampling, or simple stratified sampling (no design effect)




Comparison sampling frames

Achim Koch (GESIS)In household and address samples: selection of respondent by interviewerLeads to overrepresentation of women in household/address samplesRelated to higher response rates

Close control on who answers the questionnaire in a household even more problematic in web survey




ESS 1 – 4:Sample of individuals: Bias X response rate




ESS 1 – 4:Sample of hhs/addresses: Bias X response rate

r = .44




Nonresponse ESS

Response targets• 70% response rate• 3% noncontact rate

Standards in• Response rate calculation• Contact form• Nonresponse analysis




Response efforts and monitoring

Guidelines response enhancement• Incentives/Brochure• Refusal conversion

Call schedule requirements• Minimum number of calls• Timing of calls (evening, weekend)

Contact form• Interviewer observations• Call records




Response rates ESS (selection all rounds)

CH FR ES UK DE BE NO DK NL PT SE HU SI FI PL0

10

20

30

40

50

60

70

80

2002 2004 2006 2008 2010




Response rates over time




Nonresponse

Response rate highest for face-to-faceFunctional illiteracy (differs across countries)Nonresponse analysis• What happens in web• Noncontact, refusal, not able• Reasons for refusal• Interviewer judgment• Refusal conversion




Questionnaire design

Questionnaire length• Split questionnaire

Matrix designOver time

Question design• Unimode• Developed for face-to-face




Role interviewers

Select respondents (+)Obtain cooperation (+)Conduct interview• Explain questions (-)• Explain process (+)• Sensitive questions (+/-)• Validity answers (+)• Social desirability (-)







Respondent understands questions

Turkey

Roman

ia

Ukraine

Russia

n Fed

eratio

nIsr

ael

SpainLa

tvia

Sloven

ia

Croatia

Netherl

ands

Bulgari

a

Czech

Rep

ublic

Cyprus

Portug

al

Belgium

Greece

United

King

dom

Slovak

ia

Hunga

ry

Poland

Estonia

Sweden

Norway

Irelan

d

France

Denmark

Switzerl

and

German

y

Finlan

d0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Never Almost never Now and then Often Very often




3.5 4 4.5 52

2.5

3

3.5

4

4.5

5

5.5

6

6.5

7

Belgium

Bulgaria

CroatiaCyprus

Czech Republic

Denmark

Estonia

Finland

France

Germany

GreeceHungary

Ireland

Israel

Latvia

Netherlands

Norway

Poland

PortugalRomania

RussiaSlovakia

Slovenia

Spain

Sweden

Switzerland

Turkey

Ukraine

UK

f(x) = 2.88786152942555 x − 7.90725721440852R² = 0.424119935946303

interviewer: respondent understands questionnaire

resp

onde

nt: s

ocia

l tru

st




Measurement

Mode effectsPart of analysisNo simple adjustmentMixed modes:• Selection effects and mode effects• Always comparison face-to-face• Mode in analyses




Cross-national comparison

Comparison over time• Mode change or time change

Comparison across countries• Mode difference or cultural difference

Mixed modes within countries• Selection effect or mode effect

Mode distribution over timeFeasibility of web for ESS (length, type of questions)




Thank you for your attention


Documents

From face-to-face to web in a cross-national study