Upload
others
View
0
Download
0
Embed Size (px)
Citation preview
European Social Survey
ESS is a European Research Infrastructure Consortium (ESS ERIC)
THE EUROPEAN SOCIAL SURVEY INFRASTRUCTURE: PAST, PRESENT AND FUTURE Dr. Rory Fitzgerald Director, ESS ERIC City University London
3rd International ESS Conference Lausanne
15 July 2016
2
European Social Survey infrastructure
Looking back: the past
Where we currently are: the present
Looking forward: the future
Today’s session:
THE PAST
3
4
European Social Survey: the past
Why we needed ESS
“There is a wealth of data on individuals and households that are regularly collected by statistical offices and other administrative agencies. These data, however, are frequently not comparable across nations and are often not accessible to researchers for reasons of administrative self-interest or data protection. Absolutely crucially, however, they do not deal with the whole range of individual orientations (attitudes, beliefs and behaviours) that are central in understanding modern societies, and that can be assessed by survey research. “Very many individual surveys have been and will be conducted in particular European countries. Inevitably, however, these are tailored to individual projects and interests. The social sciences, if they are to make progress, require regular cross-national surveys that are conceptually well anchored, conducted according to rigorous methodological standards and are available at little cost to the entire research and policy community.” ESF Blueprint, March 1998
5
European Social Survey: the past
Key milestones for ESS: ESS commission funding (2001) 22 countries took part in Round 1 (2002) 36 European countries involved (2002-16) Received Descartes Prize (2005) Infrastructure funding (from 2006) Groves Review (2008) ESFRI Roadmap ERIC (2013) ESFRI landmark (March 2016)
“The panel unanimously finds that the importance of ESS, its demonstrated success in initial launch, and its clear signals of impact justify fully continuous funding at levels necessary to achieve its vision and maintain its quality.” Groves Review, 2008
THE PRESENT
6
7
European Social Survey: the present
Use of ESS infrastructure
0
10000
20000
30000
40000
50000
60000
70000
80000
90000
100000
janv..05 janv..06 janv..07 janv..08 janv..09 janv..10 janv..11 janv..12 janv..13 janv..14 janv..15 janv..16
Registered ESS Users (January 2005 - January 2016)
8
European Social Survey: the present
Data users and data sessions per month
0
500
1000
1500
2000
2500
3000
3500
4000
4500
5000
Dec 2003
Dec 2004
Dec 2005
Dec 2006
Dec 2007
Dec 2008
Dec 2009
Dec 2010
Dec 2011
Dec 2012
Dec 2013
Dec 2014
Dec 2015
Data users
Data sessions
9
European Social Survey: the present
Data users and data sessions per year
0
5000
10000
15000
20000
25000
30000
35000
40000
2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015
Data users
Data sessions
10
European Social Survey: the present
Data downloaders per year
0
2000
4000
6000
8000
10000
12000
14000
2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015
Downloaders
Trends in ESS academic publishing (Google Scholar, 2003-2014, N=2705)
1131
89
133
193210
263292
313
375 379
416
0
50
100
150
200
250
300
350
400
450
2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014
Overall Journal articles Books & chapters Conference & working papers
2705 publications overall:▪ 1281 journal articles▪ 472 books & chapters▪ 470 working papers▪ 330 conference papers
Journals with most ESS publications (2003-2014, N=1281)
11.9
2.9
5
6.3
8.1
11.5
19.2
35.1
0 5 10 15 20 25 30 35 40 45 50
Other
Media, Public opinion
Psyshology
Health, Medicine
Methods, Statistics
Economy
Political science
Sociology
▪ Social Indicators Research 60 ▪ European Sociological Review 55 ▪ European Societies 32 ▪ Journal of European Social Policy 19 ▪ International Journal of Public Opinion Research 18 ▪ Survey Research Methods 17 ▪ International Journal of Sociology 14 ▪ International Journal of Comparative Sociology 14 ▪ Comparative Political Studies 14 ▪ West European Politics 14 ▪ Journal of Cross-Cultural Psychology 13
Topics addressed in ESS publications (2003-2014, N=2705)
4053333
4795
111114116120
136139139
159185
209222227233234
282289
338600
0 100 200 300 400 500 600 700
MethodsTransition
EnvironmentEuropeNation
Media-ICTCrime
ReligionEducation
HealthGender
CitizenshipAge groups
Culture, valuesInequalities
EconomySocial capital
FamilySWB-QOL
WorkPublic policies, welfare
ImmigrationPolitics, democracy
67
71
73
83
102
199
0 100 200 300
Democratic system
Political attitudes, culture
Political protest
Political parties
Political trust
Political participation
Distribution of 1st author's affiliation across countries (2003-2014, N=2705)
3132363738
52555660616364
7172
106137
150164
245272
319363
0 100 200 300 400
RomaniaHungary
AustriaGreece
IsraelEstoniaIreland
PortugalPoland
DenmarkFinlandFrance
SwitzerlandNorway
ItalySpain
SwedenBelgium
NetherlandsUSA, Canada
GermanyUnited Kingdom
49.2
13.4
18
8.9
10.5 Europe-WestEurope- NorthEurope-SouthEurope-EastNorth America
% share of publications where a country was included (1)
(sorted according to the number of rounds fielded, 2003-2014, N=1520)
38.139.7
4852.9
55.755.856.1
6060.3
63.965.666.6
68.970.570.971.472.372.473.573.7
0 20 40 60 80 100
SlovakiaEstonia
Czech RAustria
SwitzerlandHungarySlovenia
PolandIreland
NorwayPortugal
FranceFinland
NetherlandsBelgium
SpainDenmarkSweden
United KingdomGermany 7 rounds
6 rounds
Trend of multiple round usage % share of publications where more that one ESS round was used, (2003-2014,
N=1580)
06.3
1523 25.4 26.1
34.2 33.742.4
49.8
0
10
20
30
40
50
60
70
80
90
100
<2005 2006 2007 2008 2009 2010 2011 2012 2013 2014
more than 1 round 3 rounds 4 rounds 5 rounds
18
European Social Survey: the present
Non-academic impact of the ESS Setting new standards in cross-national research Open data Methods adopted Policymakers and Government Press coverage
19
European Social Survey: the present
Non-academic impact of the ESS
HAS PRESENT MET CHALLENGES OF PAST
20
MEASUREMENT EQUIVALENCE ACHIEVED?
In each round two complex concepts in the ESS are tested for measurement equivalence, i.e. whether the same construct is being measured across countries or linguistic groups.
Testing in the ESS is done in three steps:
Configural invariance: testing evaluates whether a theoretical model holds cross-nationally, i.e. whether the factor structure is the same.
Metric invariance: testing restricts the factor loadings to be equal across groups. Its interpretation is that respondents across groups attribute the same meaning to the latent concept under study. →Only if metric invariance is established the relationship with other variables can be studied.
Scalar invariance: testing restricts (in addition) the intercepts to be equal across groups.
→ If established groups’ latent means can be compared.
Concept tested Number of indicators
Across countries / groups (combination of language and country)
Interest in Political Issues presented in the Media
3 29/30 countries (R1-R4)
Depression 8 6/27 countries (R3 and R6)
Trust in police procedural fairness
3 18/27 countries (R5)
Political satisfaction 2 10/25 (R1- R5)
Quality of state services 2 7/25 (R1 – R5)
Trust in political institutions 2 21/33 groups (R2-R5)
Trust in political authorities 2 33/33 groups (R2-R5)
Internal political efficacy 3 14/18 groups (R7 first data release)
External political efficacy 3 15/ 18 groups (R7 first data release)
Scalar invariance - Allowing for comparison of relationships and means across groups in each Round
Concept tested Number of indicators
Within country longitudinal results
Interest in Political Issues presented in the Media
3 29/30 countries (R1-R4)
Depression 8 22/22 countries (R3 and R6)
Political satisfaction 2 For most countries that participated in 4 or 5 rounds, within-country scalar equivalence established over 4-5 rounds.
Quality of state services 2 For most countries that participated in 4 or 5 rounds, within-country scalar equivalence established over 4-5 rounds.
Scalar invariance - Allowing for comparison of relationships and means across rounds within each group
Equivalence achieved? • Scalar invariance test is a strict test requiring loadings and intercepts to
be equal across groups, even though it could be established for most of the groups tested.
• Sometimes substantive researchers are more interested in relationships with other variables than in the comparison of latent means. For that metric invariance is a sufficient requirement. ESS concepts exhibit high metric invariance.
• When invariance is established the measurement of a concept is not dependent on group membership, revealing differences across countries as substantive.
• Invariance testing provides valuable information for analysts, for instance:
Common understanding of political affairs across Europe:
- Concepts asked in the ESS about political attitudes exhibit high measurement equivalence.
- ‘Interest in political issues presented in the media’ was scalar invariant across 4 rounds tested and 29 out of 30 countries
- ‘Trust in political authorities’ was scalar invariant in 4 rounds tested and 33 out of 33 groups
Interpretation of the ‘depression scale’ is not the same across European countries although it is consistent over time within countries.
THE FUTURE
25
26
European Social Survey: the future
ANALYSIS CHALLENGES Weighting
Not always applied (not clear in journals) Most common user query Complex sampling designs not accounted for Use of a harmonized methodology for post stratification weights needs to be evaluated
Interviewer effects not accounted for Differences in question quality rarely accounted for Equivalence testing not always performed The curse of the ESS total column in tables More on-line training and guidance required in future Combining data across rounds to increase sample size of sub groups – issues?
27
European Social Survey: the future
Reasons for rejecting mixed mode interviewing Synthesis report (Villar and Fitzgerald, 2016) ESS has always used face-to-face interviews Six studies into measurement, representativeness and cost efficiency Same sampling frame as the standard ESS Response rates remained lower in all mixed-mode designs compared to face-to-face design Telephone interviews had poor response rates Mixed-mode online performed better – results still disappointing Nonresponse bias may be increased by moving to mixed-mode Mode effects would be introduced but no easy way to correct for these Estonia – hope for future?
A collaboration between:
This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 654221.
www.seriss.eu @SERISS_EU
• European Social Survey (ESS ERIC)
• Survey of Health, Ageing and Retirement in Europe (SHARE ERIC)
• Consortium of European Social Science Data Archives (CESSDA AS)
• Generations and Gender Programme (GGP)
• European Values Study
• WageIndicator Survey
• Strengthen social science research across Europe and beyond by overcoming fragmentation and fostering interoperability, harmonisation and innovation
• Equip social science infrastructures to play a major role in addressing
key societal challenges and ensure that national and European policy making is built on a firm socio-economic evidence base
• Promote the value of the social sciences to the wider research
community, and policymakers
This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 654221.
www.seriss.eu @SERISS_EU
Designed to:
Achieved by:
• Addressing key challenges for cross-national data collection e.g. accurately representing the population, achieving equivalence through translation
• Breaking down barriers between social science infrastructures
via training and networking events and the development of shared online tools to facilitate harmonised data collection and documentation.
• Embracing the future of the social sciences by examining the legal
and ethical challenges associated with new forms of data, developing a cross-national probability-based web survey and exploring automated coding for socio-economic variables
This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 654221.
www.seriss.eu @SERISS_EU
This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 654221.
www.seriss.eu @SERISS_EU
32
European Social Survey: the future
Methodology: CRONOS web panel Assessing viability of obtaining accurate data online CRONOS will use ESS to recruit respondents from three countries Recruits will take part in 12-month web panel Respondents will be asked to complete a 20-minute web survey every other month at any point during a two-month period Surveys will be programmed and administered using Questback-EFS Questions will be taken from European Value Study, ESS and European Quality of Life Survey Respondents will be given a small token of appreciation Some data will be freely accessible If positive outcome, could be adopted by ESS in future
Commitment to the ESS
16 countries ESS ERIC Members (6 Guests)
Austria Portugal
Belgium Slovenia
Czech republic Sweden
Estonia Switzerland* France UK Germany (Hungary) Ireland
Lithuania (Finland) Netherlands (Iceland) Norway (Israel) Poland (Russia)**
36 Countries have taken part in at least one round of the survey:
27 EU states (not Malta)
+ 9 others:
Norway, Switzerland, Israel, Turkey, Russia, Ukraine, Iceland, Kosovo, Albania
34
European Social Survey: the future
ESS SUSTAIN Membership Development activities
Membership Development Advisor appointed Roaming Ambassadors appointed Events / seminars in target countries
Funding and Governance Working Group (including BREXIT) Impact case study Enhanced communications
Increasing media presence Launched social media presence Updating News section on ESS website Working with national teams to promote ESS in media across Europe
Funded by Commission (H2020)
35
European Social Survey: conclusions
Clear success of the ESS has been demonstrated and the current model is sustainable Have to keep making the case for rigour Need to further increase visibility of the project Use of data and comparability findings suggest ESS is a leader in the field Mixed mode would undermine quality but FtoF fieldwork costs rising: web additions in the future? ESS core questionnaire will be reviewed Rotating modules continue to allow new topics to be included SERISS project offers opportunities for joint methodological development Especially CRONOS panel SUSTAIN project helping to secure the future but challenging landscape BREXIT is a new challenge