From face-to-face to web in a cross-national study
Ineke StoopSCP/CST
www.europeansocialsurvey.org4th International Workshop on Internet Survey Methods,
Statistics Korea,2012
ESSAcademic survey in 30+ European countriesAttitudes, values and beliefsBiennial, start in 20021+ hour, face-to-face, effective sample size ~ 1500Funders• National Science organisations• European Commission, Framework Programmes• European Science Foundation
Management model• Central coordination• National implementation
www.europeansocialsurvey.org4th International Workshop on Internet Survey Methods,
Statistics Korea,2012
Aims ESS
To monitor and interpret public attitudes and values within Europe and to investigate how they interact with Europe’s changing institutions
• Provide data on beliefs, attitudes and values for scientific and policy making purposes
• Measure attitude change in a changing Europe
To advance and consolidate improved methods of cross-national survey measurement in Europe and beyond
• Collect data according to highest standards
• Generate methodological research
• Develop and disseminate new best practices
• Develop and improve social survey research infrastructure in Europe
www.europeansocialsurvey.org4th International Workshop on Internet Survey Methods,
Statistics Korea,2012
Content questionnaires ESS
Every roundSocio-demo-graphic items
• household composition• education• housing• occupation• social structure
Attitudes and behaviour patterns• religious affiliation and identity• ethnic and national identity• political and social trust• party affiliation and voting
behaviour• media consumption• value orientations• social exclusion
Round specificR1 Immigration
Citizenship, involvement and democracy
R2 Family, work and well-beingOpinions on health and
care- seekingEconomic Morality
R3 Personal & Social Well-beingThe Timing of Life
R4 AgeismWelfare attitudes
R5 Family, work and well-being in times of economic recession
Trust in police and courts
R6 Personal & Social Well-beingUnderstanding, evaluation democracy
www.europeansocialsurvey.org4th International Workshop on Internet Survey Methods,
Statistics Korea,2012
Access ESS
Round 1-5 availableFully documentedFree accessTables and graphics online, or download dataTraining package for young researchers
Everything availableAll deviations documented
www.europeansocialsurvey.org4th International Workshop on Internet Survey Methods,
Statistics Korea,2012
ESS top data user countries (>50) (June 2012) (N=48,973)
Brazil
Korea, Republic of
Iran, Islamic Republic of
Iceland
Cyprus
Philippines
Bulgaria
Latvia
Turkey
Romania
Ukraine
Israel
Estonia
Sweden
Finland
Austria
Switzerland
France
Poland
Spain
Slovenia
Belgium
0 500 1000 1500 2000 2500 3000 3500 4000 4500 5000
www.europeansocialsurvey.org4th International Workshop on Internet Survey Methods,
Statistics Korea,2012
Participation R1 R2 R3 R4 R5
AustriaBelgiumBulgariaCroatiaCyprusCzech Denmark
EstoniaFinlandFrance
GermanyGreeceHungaryIcelandIrelandIsraelItalyLatviaLithuaniaLuxembourgNetherlandsNorway
PolandPortugalRomaniaRussiaSlovakiaSloveniaSpainSwedenSwitzerlandTurkeyUK UkraineTotal 22 26 25 31 28
www.europeansocialsurvey.org4th International Workshop on Internet Survey Methods,
Statistics Korea,2012
Methods: high quality
Strictly random sampling• No substitution, no proxy
Population: all inhabitants aged 15+High target response rates• 70% target response rate, max. 3% noncontact
Face-to-face fieldwork• Gold standard, high response rates, universally applicable• Increasingly expensive• Rare in some countries
Pilots, pretesting and fieldwork monitoringAnalysis of cross-national differences• Nonresponse bias, measurement error, sampling design effects
www.europeansocialsurvey.org4th International Workshop on Internet Survey Methods,
Statistics Korea,2012
Methods: central coordination
Central specifications for fieldworkClose monitoring national fieldwork (in advance and during fieldwork)Sampling designTranslation proceduresGuidelines response enhancementGuidelines interviewer trainingContact forms, response calculationContext, events
www.europeansocialsurvey.org4th International Workshop on Internet Survey Methods,
Statistics Korea,2012
Can we compare countries?
Survey design, methodology,
implementation
Attitudes, values, beliefs
Survey results
www.europeansocialsurvey.org4th International Workshop on Internet Survey Methods,
Statistics Korea,2012
Harmonisation strategies (Körner and Meyer, 2005)
Harmonised concept
Harmonised concept
Harmonised concept
Measurementprocedure
Measurementprocedure
Measurementprocedure
National survey/Specific concept
National survey/Specific concept
National survey/Specific concept
Input harmonisation Ex-ante outputharmonisation
Ex-post outputharmonisation
www.europeansocialsurvey.org4th International Workshop on Internet Survey Methods,
Statistics Korea,2012
Output harmonisation
Ex post output harmonisation• Use existing sources
Ex ante output harmonisation• Set up new survey (or develop new instrument)
Output harmonisation• Best national quality• Or, national survey tradition• But, what about optimal comparability
www.europeansocialsurvey.org4th International Workshop on Internet Survey Methods,
Statistics Korea,2012
Output harmonisation: problems
• Interview mode• Context question• Lay-out questionnaire• Question text• Introductory text• Answer scales• Handling don’t know• Size of boxes• Use of illustrations
www.europeansocialsurvey.org4th International Workshop on Internet Survey Methods,
Statistics Korea,2012
Input harmonisation
Input harmonisation• Design new survey• Optimal comparability
Or, are identical methods and instruments really equivalent in different countries?• But, what about optimal quality
www.europeansocialsurvey.org4th International Workshop on Internet Survey Methods,
Statistics Korea,2012
From face-to-face to web in the ESS
Mixed mode research and experimentsDifferent survey modes over timeDifferent survey modes across countriesMixed mode within countries: different composition over timeWhich modes:• Face-to-face• Telephone• Web
www.europeansocialsurvey.org4th International Workshop on Internet Survey Methods,
Statistics Korea,2012
ESS studies
Martin, Peter (2011) A Good Mix? Mixed Mode Data Collection and Cross-national Surveys. ASK Research & Methods, Vol. 20 (1, 2011) 5–26.Martin, Peter (2011) What makes a good mix? Chances and Challenges of Mixed Mode Data Collection in the European Social Survey. CCSS Working Paper No. 02.Martin, Peter, and Lynn, Peter (2011) The Effects of Mixed Mode Survey Designs on Simple and Complex Analyses. CCSS Working Paper No. 02.Roberts, Caroline (2007) Mixing Modes of Data Collection in Surveys: A Methodological Review. ESRC National Centre for Research Methods. NCRM Methods Review Papers, NCRM/008.
www.europeansocialsurvey.org4th International Workshop on Internet Survey Methods,
Statistics Korea,2012
From face-to-face to web in the ESS
CoverageSamplingNonresponseQuestionnaire designRole interviewersMeasurementAnalysisCross-national comparison
www.europeansocialsurvey.org4th International Workshop on Internet Survey Methods,
Statistics Korea,2012
Coverage
www.europeansocialsurvey.org4th International Workshop on Internet Survey Methods,
Statistics Korea,2012
Internet access and broadband connections in households, EU27 (%)
2006 2006.5 2007 2007.5 2008 2008.5 2009 2009.5 2010 2010.5 20110
10
20
30
40
50
60
70
80
Internet access Broadband connections
www.europeansocialsurvey.org4th International Workshop on Internet Survey Methods,
Statistics Korea,2012
Individuals who used the internet at least once a week, 2011 (%)
IS SE NO NL DK LU FIUK BE DE AT FR EE SK IE
EU27 LV HU MT SICZ ES LT PL CY IT PT GR BG RO
0
10
20
30
40
50
60
70
80
90
100
www.europeansocialsurvey.org4th International Workshop on Internet Survey Methods,
Statistics Korea,2012
Individuals who used the internet on average at least once a week, by sex, age and education, EU27, 2011 (%)
All individuals
Men Women 16-24 years
25-54 years
55-74 years
Low education
Medium education
High education
0
20
40
60
80
100
www.europeansocialsurvey.org4th International Workshop on Internet Survey Methods,
Statistics Korea,2012
Sampling principles
Identical population definitionComplete coverage• No upper age limits• (Institutional population)• (Non-native language speakers)
Probability samplingEffective sample sizeOptimal sampling design in each country• Address, household, individual• Address and household: selection of target respondent in the field
www.europeansocialsurvey.org4th International Workshop on Internet Survey Methods,
Statistics Korea,2012
ESS R2 sampling frames (Häder and Lynn)Country Frame RemarksAustria Telephone book Additional non-telephone households sampled in the fieldBelgium National registerCzech Republic Address register UIR-ADR Select streets, followed by field enumerationDenmark Danish Central Person RegisterEstonia Population registerFinland Population registerFrance None Area-based samplingGermany Registers from local residents’ registration
officeGreece None Area-based samplingHungary Central registryIceland National registerIreland National Electoral RegisterLuxembourg Social Security RegisterNetherlands Postal address listNorway Population registerPoland National register of CitizensPortugal None Area-based samplingSlovakia Central Register of CitizensSlovenia Central Register of PopulationSpain Continuous CensusSweden Register of populationSwitzerland Telephone registerTurkey Cluster of addressesUK Postal address listUkraine None Area-based sampling
Simple random sampling, or simple stratified sampling (no design effect)
www.europeansocialsurvey.org4th International Workshop on Internet Survey Methods,
Statistics Korea,2012
Comparison sampling frames
Achim Koch (GESIS)In household and address samples: selection of respondent by interviewerLeads to overrepresentation of women in household/address samplesRelated to higher response rates
Close control on who answers the questionnaire in a household even more problematic in web survey
www.europeansocialsurvey.org4th International Workshop on Internet Survey Methods,
Statistics Korea,201225
ESS 1 – 4:Sample of individuals: Bias X response rate
www.europeansocialsurvey.org4th International Workshop on Internet Survey Methods,
Statistics Korea,201226
ESS 1 – 4:Sample of hhs/addresses: Bias X response rate
r = .44
www.europeansocialsurvey.org4th International Workshop on Internet Survey Methods,
Statistics Korea,2012
Nonresponse ESS
Response targets• 70% response rate• 3% noncontact rate
Standards in• Response rate calculation• Contact form• Nonresponse analysis
www.europeansocialsurvey.org4th International Workshop on Internet Survey Methods,
Statistics Korea,2012
Response efforts and monitoring
Guidelines response enhancement• Incentives/Brochure• Refusal conversion
Call schedule requirements• Minimum number of calls• Timing of calls (evening, weekend)
Contact form• Interviewer observations• Call records
www.europeansocialsurvey.org4th International Workshop on Internet Survey Methods,
Statistics Korea,2012
Response rates ESS (selection all rounds)
CH FR ES UK DE BE NO DK NL PT SE HU SI FI PL0
10
20
30
40
50
60
70
80
2002 2004 2006 2008 2010
www.europeansocialsurvey.org4th International Workshop on Internet Survey Methods,
Statistics Korea,2012
Response rates over time
www.europeansocialsurvey.org4th International Workshop on Internet Survey Methods,
Statistics Korea,2012
Nonresponse
Response rate highest for face-to-faceFunctional illiteracy (differs across countries)Nonresponse analysis• What happens in web• Noncontact, refusal, not able• Reasons for refusal• Interviewer judgment• Refusal conversion
www.europeansocialsurvey.org4th International Workshop on Internet Survey Methods,
Statistics Korea,2012
Questionnaire design
Questionnaire length• Split questionnaire
Matrix designOver time
Question design• Unimode• Developed for face-to-face
www.europeansocialsurvey.org4th International Workshop on Internet Survey Methods,
Statistics Korea,2012
Role interviewers
Select respondents (+)Obtain cooperation (+)Conduct interview• Explain questions (-)• Explain process (+)• Sensitive questions (+/-)• Validity answers (+)• Social desirability (-)
www.europeansocialsurvey.org4th International Workshop on Internet Survey Methods,
Statistics Korea,2012
www.europeansocialsurvey.org4th International Workshop on Internet Survey Methods,
Statistics Korea,2012
Respondent understands questions
Turkey
Roman
ia
Ukraine
Russia
n Fed
eratio
nIsr
ael
SpainLa
tvia
Sloven
ia
Croatia
Netherl
ands
Bulgari
a
Czech
Rep
ublic
Cyprus
Portug
al
Belgium
Greece
United
King
dom
Slovak
ia
Hunga
ry
Poland
Estonia
Sweden
Norway
Irelan
d
France
Denmark
Switzerl
and
German
y
Finlan
d0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Never Almost never Now and then Often Very often
www.europeansocialsurvey.org4th International Workshop on Internet Survey Methods,
Statistics Korea,2012
3.5 4 4.5 52
2.5
3
3.5
4
4.5
5
5.5
6
6.5
7
Belgium
Bulgaria
CroatiaCyprus
Czech Republic
Denmark
Estonia
Finland
France
Germany
GreeceHungary
Ireland
Israel
Latvia
Netherlands
Norway
Poland
PortugalRomania
RussiaSlovakia
Slovenia
Spain
Sweden
Switzerland
Turkey
Ukraine
UK
f(x) = 2.88786152942555 x − 7.90725721440852R² = 0.424119935946303
interviewer: respondent understands questionnaire
resp
onde
nt: s
ocia
l tru
st
www.europeansocialsurvey.org4th International Workshop on Internet Survey Methods,
Statistics Korea,2012
Measurement
Mode effectsPart of analysisNo simple adjustmentMixed modes:• Selection effects and mode effects• Always comparison face-to-face• Mode in analyses
www.europeansocialsurvey.org4th International Workshop on Internet Survey Methods,
Statistics Korea,2012
Cross-national comparison
Comparison over time• Mode change or time change
Comparison across countries• Mode difference or cultural difference
Mixed modes within countries• Selection effect or mode effect
Mode distribution over timeFeasibility of web for ESS (length, type of questions)
www.europeansocialsurvey.org4th International Workshop on Internet Survey Methods,
Statistics Korea,2012
Thank you for your attention