Measurement Quality in Social Networks - Pàgines de la UABgrupsderecerca.uab.cat/egolab/sites/grupsderecerca.uab.cat.egolab/files/Coromina... · 1 Universitat de Girona Measurement

1

Universitat de Girona

Measurement Quality in Social Networks

LLuís Coromina

Quantitative Methods, University of Girona Department of Economics,

Campus Montilivi, 17071 Girona, Spain E-mail: [email protected]

2Universitat de Girona

Outline

• Studies on Social Network Data Quality

• Survey Design o Data collection o Questionnaire Design

• Types of hypothesis tested in other Quality studies

• Quality Measurement (Validity and Reliability)

• Measurement procedure o Multitrait-Multimethod Model (MTMM) o Multilevel MTMM o Interpretation


Outline

• INSOC PROJECT: o Social Network Data collection Design o Multitrait-Multimethod Model results o Meta-analysis Design o Meta-analysis Results

• Results and comparison from others studies.

Intro

4

Objective


4

Main purpose of scientific research is the interpretation and/or prediction

of phenomena

Quality measurement instruments (empirical data) is of crucial importance

It is important to know how well we can measure with certain

measurement instruments and how much a researcher can rely on findings

based on data collected.

Intro

5

Objective


5

Egocentered network data are especially important.

Network characteristics (network size, structure and composition)

Characteristics of network members (feeling of closeness or importance;

frequency of advice, collaboration, social support) collected from the

respondent (ego).

Intro


Quality in Social Network Survey Data

6

Studies on Social Network Data Quality

Killworth & Bernard, 1976, 1980; Bernard et al., 1979,1980, 1982;

Hammer, 1984; Sudman, 1985, 1988; Freeman and Romney, 1987;

Freeman, Romney & Freeman, 1987; Corman and Bradford, 1993;

Hlebec, 1993; Brewer and Webster, 1997,…

Intro



7

People are generally very inaccurate in reporting on their past

interactions with other people.

People remember long-term or typical patterns of interaction with other

people rather well.

Accuracy of reporting about interactions is also influenced by the

frequency of interaction (more frequent contact with group members,

more accurate reports) and by the reliability of an individual

respondent on about actual interactions (Romney and Faust, 1982; Romney

and Weller, 1984).

Intro



8

Quality of measurement, specially reliability, can be affected by the type of

social support (Ferligoj & Hlebec, 1999).

+ Emotional and informational support dimensions

Reliability

- Social companionship and material support dimensions

Reasons for ‘higher quality’

More “intimate” types of support � more important to the respondents.

� Relatively small number of very

important people.

Less effortful for the respondent. Network

members are important and close

Intro


Data Collection Methods

9

DATA COLLECTION METHODS

In Social Sciences, the most frequent measurement instrument is a SURVEY

Intro



10

Intro



11

With interviewer Self-administration

Paper •••• Personal (Face-to-face)

•••• telephone

•••• “paper self-

administered”

•••• mail

Electronic •••• computer assisted personal

interview (CAPI)

•••• computer assisted telephone

interview (CATI)

•••• E-mail

•••• Web surveys

•••• Computer Assisted Self-

Interview (CASI)

•••• ACASI

•••• Mobile phone surveys

NO ONE COLLECTION METHOD IS BEST FOR ALL CIRCUMSTANCES.

THE CHOICE MUST BE MADE WITHIN THE CONTEXT OF THE PARTICULAR

OBJECTIVES OF THE SURVEY AND THE RESOURCES AVAILABLE.

Intro



12

Characteristics of the Population

+

Characteristics of The Sample

+

Types of Questions

+

Question Topic

+

Response Rate

+

€€ Cost $$

+

Time

Intro



13

PERSONAL INTERVIEWING

ADVANTAGES: DISADVANTAGES:

Generally: highest cooperation and lowest

refusal rates

Most costly mode of administration

Allows for longer, more complex interviews Longer data collection period

High response quality Interviewer concerns

Takes advantage of interviewer presence

Multi-method data collection

TELEPHONE INTERVIEWING


Less expensive than personal interviews Samples of general population Non-response

Shorter data collection period than personal Questionnaire constraints

Interviewer administration (vs. mail) Difficult to administer questionnaires on

sensitive or complex topics

Better control and supervision of interviewers

(vs. personal)

Biased against households without

telephones, unlisted numbers

Better response rate than mail for list samples

Intro



14

MAIL SURVEYS


Generally lowest cost Most difficult to obtain cooperation

Can be administered by smaller team of

people (no field staff)

No interviewer involved in collection of data

Access to otherwise difficult to locate, busy

populations

More likely to need an incentive for

respondents

Respondents can look up information or

consult with others

Need good sample

Slower data collection period than telephone

Intro



15

WEB SURVEYS

ADVANTAGES: DISADVANTAGES

Lower cost (no paper, postage, mailing,

data entry costs)

% of homes own a computer;

% have home e-mail.

Can reach international populations Representative samples difficult - cannot

generate random samples of general

population

Time required for implementation reduced Differences in capabilities of people's

computers and software for accessing

Web surveys

Complex skip patterns can be programmed Different ISPs/line speeds limits extent of

graphics that can be used

Sample size can be greater

Intro



16

MEASUREMENT QUALITY IMPLICATIONS OF MODE SELECTION

Social Desirability

� Under-represent (illegalities) or Over-represent (voting) different population groups.

� Sensitive information from respondents

� Self-administered methods produce fewer social desirability effects.

Response effects (wording, answer categories order, or questions order)

Response order (Primacy-visually- and recency –telephone)

Data completeness (with interviewers more questions answered)

Completion times

Intro



17

Mean completion times(in seconds)

by mode (web vs Telephone) and age group

Intro



18

Differences in data quality for different methods????

Data collection method for ego-centered network data.

Most used face-to-face data collection mode.

Face to face vs. Telephone (Kogovšek et al., 2002)

Validity and Reliability of web surveys

(Couper, 2000; Dillman, 2000; Couper et al., 2001; Vehovar et al., 2002)

Quality of Egocentered Network Data Collected via Web

Marin (2002) ; Lozar Manfreda et al. (2004).

Intro


19

Questionnaire Design

QUESTIONNAIRE DESIGN

Problems in the response process Survey ERRORS

Krosnick (1999); Groves (2004),

Tourangeau, Rips & Rasinski (2000), Dillman, (2007)

Intro


20


1) Failure to encode the information required

Problem of asking people about what topics that they do not pay much

attention OR are not important.

Improvement: Pre-test; Filter questions (or multiple name generator)

2) Misinterpretation of the questions

Hospital study: During the past 12 months, since _______, how many times have you

seen or talked to doctor [NAME i ] about your health? Do not count any time you might

have seen a doctor while you were a patient in a hospital, but count all other times your

actually saw or talked to a medical doctor of any kind.

Intro


21


3) Forgetting and other memory problems

Example:

Low options High responses

Responses % Responses %

<1/2 hr 7.4 < 2 ½ hr 62.5%

½ to 1hr 17.7 2 ½ to 3 hr 23.4%

1 to 1 ½ hr 26.5 3 to 3 ½ hr 7.8%

1 ½ hr to 2 hr 14.7 3 ½ hr to 4 hr 4.7%

2 to 2 ½ hr 17.7 4 to 4 ½ hr 1.6%

>2 ½ hr 16.2 >4 ½ hr 0%

Question about television watching can go from 16% to 37% saying that they watched

more than 2.5 hours/day, depending on the set categories.

Intro


22


4) Sensitive (social desirability) questions (recommendations)

• More time to respondents to answer in order to not misreport information (Krosnick 1999)

• Question example for strategy for over-reported behaviors:

In talking about elections, we often find that a lot of people were not able to vote because they are not registered, they were sick, or they just didn’t have the time. How about you – did you vote in the elections this November?

• Use self-administration, computer-administered questionnaire or similar methods to improve reporting.

Intro


23


5) Failure to follow instructions (mainly for self-administered respondents)

If first page is half-page explaining instructions, respondent use to skip that part.

Intro


24


Instructions must be exactly where is needed and not in a separate section

Intro


25


Format of response scale (Network Questionnaires)

• Binary judgments (least difficult for respondents) about whether

respondents have a specified relationship with each actor on the roster.

• Ordinal ratings of tie strength

• Rankings: More demanding for respondents.

Majority of respondents preferred binary over ranking or rating…

Intro


26


…. BUT reliability for ratings > binary judgments (Ferligoj & Hlebec, 1999).

Recommendations (not only network questionnaires)

• Use closed questions for measuring attitudes

• Use 5 point response scale and labeled every scale point. With 9 or 11

point response scale the beginning and the ending.

• Use ranking (preferences order) only if the respondents can see all the

alternatives: otherwise use paired comparisons.

Intro


27


Name Generator Design

Studies suggest using minimal modules of name generators:

Van Der Poel (1993):

3-item: Discussing a major life change, aid with household tasks, and monthly

visiting.

5-items, add: Borrowing household money and going out socially.

Bernard et al. (1990):

5 items: Social activities, hobbies, personal problems, advice about important

decisions, closeness

Burt (1997):

3-item: GSS important matters issue, socializing, and discussion of a job

change

Intro


28


Clusters of persons named by relationships (families, workplaces,…) than by

similarity of individual features -gender, race or age- (Fiske ,1995)

Use Recognition rather than Recall (when possible) - Level of ‘forgetting’-

(Brewer, 2000)

Multiple name generators may limit forgetting because persons forgotten for one

generator are often name in response to others.

Single name generator maybe sufficient for ‘core networks’

Intro


Quality in Social Network Data

29

Measurement Comparisons

Questionnaire components

Network ties

Intro



30

In summary,

Validity and Reliability of survey data, social networks, can be affected by

many characteristics of the measurement instrument ( Name Interpreters).

a) Response scale (binary; 5 ordinal ending; 5 ordinal all; line drawing scale; 11

ordinal)

b) Response category labels( all categories or end points).

c) Name Generator method (Recognition vs free recall).

c) Data collection method (face-to-face interview, telephone, web survey,…)

e) Question wording (after we obtain the list of alters with name generators,

we can ask name interpreter questions “by alters” or “by questions”).

f) Layout of the questionnaire: Plain vs Graphical

Intro



31

Types of networks analyzed:

Social support (Weiss, 1974; Hirsch, 1980; Wills, 1985:, vaux, 1988)

1) instrumental support (aid, material support, or tangible support) is the

provision of financial aid, material support, and necessary services),

2) informational support (advice support, appraisal support, or cognitive guidance)

involves help in defining, understanding, and coping with problematic events)

3) emotional support (or close support),

4) social companionship (diffuse support and belongingness) relates to time spent

with others in leisure activities).

Intro



32

I) Frequency of Scientific advice II) Frequency of Collaboration III)Asking for crucial Information IV) Social activities outside the work

Tie Characteristics: Demographic variables

a) Measures of tie strength Education Frequency of contact Gender Feeling of closeness Age

b) Feeling of importance

c) Frequency of the alter

upsetting the ego

Intro


Hypothesis and Research Questions from Studies

33


Intro



34

• To test whether the recognition data collection technique is more stable than that of free recall (Ferligoj & Hlebec,1999)

• The size of the social network and age of the respondent should not

affect the similarity of free-recall and recognition methods.

• Reliability and validity of the measurement of tie characteristics for emotional and informational support alters would be higher compared to that for social companionship and material support alters. Kogovsek & Ferligoj (2004)

• The stability of social support : Emotional support (close and important ties) more stable than Material support (provided by specialized sources).

Closeness is not required for providers of material support and informational support. Hlebec & Ferligoj (2002)

Intro



35

Hypothesis Predictors Kogovsek & Ferligoj (2005)

Measurement instrument Higher quality of measurement with by alters than by questions data collection

technique Higher quality of measurement of cognitively demanding questions with the

personal interview/by alters Higher quality of measurement of sensitive questions with the telephone

interview/by alters Network size Higher quality of measurement in small networks Personal characteristics Higher quality of measurement among female respondents Higher quality of measurement among younger respondents Higher quality of measurement among more educated respondents Personality characteristics Higher quality of measurement among extraverts

Intro



36

• We might expect that every tie or alter characteristic would be more reliably and more validly measured when the question is posed by alters than by questions ( Kogovšek , Ferligoj , Coenders , Saris, 2002)

We therefore expect that cognitively more demanding questions would be more prone to measurement errors in the telephone than in the face-to-face mode, therefore:

• It is expected that demanding questions would be more reliably and more validly measured in face-to-face interviews than by telephone

Intro

37

Quality of measurement


37

How to Measure

Intro

38



38

Reliability: The extent to which any questionnaire, test or measure

produces the same results on repeated experiments. Random error.

Validity: The extent to which any measure measures what is intended to

measure (Carmines & Zeller, 1979:12). Systematic error, bias.

Convergent validity refers to common trait variance and is inferred

from large and statistically significant correlations between measures of the

same trait using different methods.

Discriminant validity refers to the distinctiveness of the different traits;

it is inferred when correlations among different traits are less than one.

Intro

39



39

Test–retest stability of networks comparing data collection techniques of network data (Example: Free recall vs Recognition method) The test is performed twice. Giving a group of participants the same questionnaire on two different occasions, the test-retest reliability is the correlation between separate administrations of the test is high.

Intro

40



40

Q

U

A

L

I

T

Y

Multilevel

Multitrait

Multimethod

Data collection


Reliability and validity estimates by country

Meta-analysis design (MCA)

Meta Analysis results

Intro

41

Multitrait Multimethod model


41

The MTMM model has rarely been used for measurement quality assessment in social network analysis. Hlebec (1999), Ferligoj and Hlebec (1999), Kogovšek et al. (2002), egocentered networks (TS model).

For model identification purposes the MTMM approach usually requires at least 3

repeated measurements of the same variable, using three different methods

(Kenny, 1976) � Burden on the respondent and increases the cost of data

collection.

To reduce these problems:

Split ballot MTMM experimental design (Saris, 1999; Saris & Coenders,

2000) combinations of just two repetitions (methods).

Intro

42



42

Reliability based on the classical test theory (Lord & Novick, 1968):

Yij = Sij + eij e Y S

• Yij is the response of variable i measured by method j.

• Sij is called the True Score (Saris & Andrews, 1991)

• eij is the random error, related to lack of reliability.

True Score is the result of Trait and Method:

Sij = mij Mj + tij Ti

Intro

43



43

T1= Trait / variable of interest M = reaction to S= True Score the method Y = observed response e = random error

• The strength of the relationship between T and S is called Validity

Validity = 1- var (M)

Intro

44



44

Confirmatory Factor Analysis (CFA) specification of the MTMM model:

Traits (Coromina, Coenders, Ferligoj, 2006): Yij = mij Mj + tij Ti + eij

Trait 1: Ask for scientific advice

Trait 2: Collaboration with your colleagues concerning research

Trait 3: Ask for information /data/software

Trait 4: Social activities outside the work

Intro

45



45

Assumptions (Andrews, 1984):

Cov(Ti,eij)=0 ∀ij

Cov(Mj,eij)=0 ∀ij

Cov(Ti ,Mj)=0 ∀ij

It makes possible to decompose the variance of Yij

Trait variance tij2Var(Ti)

method variance mij2Var(Mj)

random error variance Var(eij)

Intro

46



46

Y11 Y21 Y31 Y41 Y22 Y12 Y32 Y42

M1 M2

T1 T2 T3 T4

e11

e21 e31 e41 e12 e22 e32

e42

Intro

47



47

Reliability increases not only with true or trait variance, but also with method variance, which also belongs to the stable or repeatable part of the measurements.

)(

)(

ij

ij

YVar

SVar

= )(

)()( 22

ij

iijjij

YVar

TVartMVarm +

Validity, % of variance of the True Score explained by the Trait:

)()(

)(

)(

)(22

22

iijjij

iij

ij

iij

TVartMVarm

TVart

SVar

TVart

+=

QUALITY, strength of the relationship between Y and T

= RELIABILITY * VALIDITY

Intro


Multilevel MTMM

48

Egocentered network data Two-level MTMM models.

The mean centred individual scores for group “g” and individual “k” can be decomposed into:

Ego (g) Between group component (g)

Alter1 Alter

2 … Alter

n Within group component (k)

YYY gkgkT −=

YYY ggB−=

ggkWgk YYY −=

Intro


Multilevel MTMM

49

WITHIN BETWEEN

Between level: Units of measurement are egocentered networks as a whole,

average across all egocentric networks � Quality of averages studied.

Within level: Individual ego-alter

Intro


Multilevel MTMM

50

Decomposition of the sample covariance matrix.

Total population covariance matrix ΣΣΣΣT

ΣΣΣΣT = ΣΣΣΣB + ΣΣΣΣW

Analysis of each component separately:

Yij = mij Mj + tij Ti + eij

Yij = mBij MBj + tBij TBi + eBij + mwij Mwj + twij Twi + ewij

YBij YWij

Intro


Interpretation

51

We obtain two reliabilities and two validities for each trait-method combination:

Between and within groups

Groups are respondents Different interpretation

Individuals are stimuli evaluated by respondents from standard way

The between-group reliabilities and validities

Quality of the measurement of the egocentered network as a whole (average).

If the ego is the focus of interest and these averages are used as data instead of

the raw responses regarding individual alters � Between-group measurement

quality is the relevant one to look at.

Intro


Interpretation

52

The within-group reliabilities and validities

Each subject is a separate unit of analysis and thus variance is defined across

stimuli presented to the same subject, not across subjects.

If the relationship is the focus of interest� within group measurement quality.

We know the between and within scores add to a total score:

Yij = mBij MBj + tBij TBi + eBij + mwij Mwj + twij Twi + ewij

YBij YWij

Then…

Intro


Interpretation

53

It is possible to compute percentages of variance.

Decomposition of variance:

Var(Yij) = mwij2Var(MWj) + mBij

2Var (MBj) +

twij2Var(TWi) + tBij

2Var(TBi)+

Var(ewij) + Var(eBij)

From the decomposition one can:

• compute overall reliabilities and validities:

By aggregating all trait, method, and error components.

• compute overall percentages of within and between variance:

By aggregating all within components and all between components.

Intro


Interpretation

54

• compute the percentage of between and within trait variance over the

total trait variance. A higher within percentage shows a higher alter variability

in the tie characteristics within a network and a lower variability of average

values of the tie characteristics between networks.

• compute a percentage of pure random error variance (i.e., within

error variance) over the total variance of the observed variables (grand total,

i.e., including all 6 components). The percentage of total variance explained by

any of the other 5 components can be computed in a similar way.

Intro

55

INSOC Project


55

INSOC project

International Network on Social Capital and Performance

Intro

56

INSOC Project


56

• Explain the academic PhD students success based on social capital.

• Develop and refine data collection methods.

• Develop social network analysis methods.

Intro

57

INSOC Project


57

Web Survey design

• Verbal (words and numbers) + rich visual.

• Special navigational features (progress indicators), animations…

• Lower costs

• Faster data collection and data analysis process

• Easy and fast questionnaire modification.

• Response when convenient

• Piping: Assigning questionnaire items based on earlier answers from the

respondent

Intro

58

INSOC Project


58

Coverage error !!!

Population with nearly universal internet access (PhD students)

Intro

59

INSOC Project


59

INSOC Questionnaire Design

Focus groups

Comparable versions (Catalan, Slovenian and Dutch)

Two independent translations

Pre-test

Intro


Population, sample and data collection

60

PhD students at the Universities of Girona, Ljubljana and Ghent

April 2003: To know PhD students’ promoters

Personal interview to promoters � name generator questions

1. Name all the teaching assistants (or doctoral assistants) whose research is mainly under your supervision.

2. Name all the researchers of whom you are formally the mentor and who work on or participate in a

research project.

3. Name your colleague professors, senior researchers, junior researchers or people working in the private

sector with whom you substantially work together on those research projects in which PhD student X [name

PhD student] is involved.

Intro



61

November 2003: e-mail invitations (86 in Girona, 191 in Ljubljana and 233

in Ghent).

Follow-up: after 1, 4 and 8, weeks

PhD responses Method 1

2weeks PhD responses Method 2

Girona 67 78% 61 91% Ljubljana 118 62% 81 69% Ghent 198 85% 55* 60%

Method 2: Different question order, different style of response category

labels and different graphical display and lay-out of the questions.

Intro



62

0%10%20%30%40%50%60%70%80%

srv

Res

pons

es

Supervisors

PhD Students

1st reminder(2/12/03)

2nd reminder(23/12/03)

3rd reminder(23/01/04

to 26/01/04)

Intro


Traits & Methods

63

Frequency Traits

• Trait 1: Ask for scientific advice

• Trait 2: Collaboration with your colleagues concerning research

• Trait 3: Ask for information/data/software

• Trait 4: Social activities outside the work

sample repetition Factor 1 Factor 2 Factor 3 Girona Only one Main questionnaire by questions all labels plain Girona Only one Follow-up by alters end labels plain

Kogovsek & Ferligoj (2004, 2005)

Group N First interview Second interview

1 320 Face-to-face/by alters Telephone/by alters

2 311 Face-to-face/by alters Telephone/by questions

3 402 Telephone/by alters Telephone/by questions

Intro


Traits & Methods

64

Intro


Traits & Methods

65

Intro


Traits & Methods

66

Example of Method 2 in Girona

Intro

67

MTMM Results -- University of Girona


67

Advice Collaboration Info Social act

within trait variance M1 67.5% 65.3% 55.6% 67.0%M2 66.1% 59.3% 47.7% 42.4%within method variance M1 0.0% 0.0% 0.0% 0.0%M2 9.6% 9.2% 12.3% 13.1%within error variance M1 13.3% 17.2% 17.3% 14.5%M2 7.0% 12.3% 16.3% 25.2%between trait variance M1 12.2% 13.1% 18.5% 11.4%M2 12.0% 11.9% 15.9% 7.2%between method variance M1 0.4% 0.4% 0.6% 0.9%M2 5.2% 5.0% 6.7% 7.1%between error variance M1 6.6% 3.9% 8.0% 6.3%M2 0.0% 2.3% 1.2% 4.9%

Goodness of fit: Yuan-Bentler χ2= 103.1 with 33df. TLI: .945

Intro

68

MTMM Results -- University of Girona


68

% of within trait variance over all trait variance. The results show that most of the error free variance corresponds to the within level. This means that egos really discriminate between the different alters.

T1 T2 T3 T4 twij

2Var(TWi)/ [ twij2Var(Twi) + tBij

2Var(TBi)] 84.6% 83.3% 75.0% 85.4%

Within level Between level Overall level T1 T2 T3 T4 T1 T2 T3 T4 T1 T2 T3 T4 Reliability coef. M1 .91 .89 .87 .91 .81 .88 .84 .81 .90 .89 .86 .89 M2 .96 .92 .89 .83 1.00 .94 .98 .86 .96 .92 .91 .84 Validity coef. M1 1.00 1.00 1.00 1.00 .98 .98 .98 .96 1.001.001.00.99 M2 .93 .93 .89 .87 .83 .84 .84 .71 .92 .91 .88 .84 Quality coef. M1 .91 .89 .87 .91 .79 .86 .82 .78 .90 .89 .86 .88 M2 .89 .86 .79 .72 .83 .79 .82 .61 .88 .83 .80 .71

Intro

69

Meta analysis Design


69

Meta-Analysis Design and Results

Intro

70



70

Summarizing the results of studies carried out by the universities of Girona, Ljubljana and Gent.

University sample repetition Factor 1 Factor 2 Factor 3 Girona (Spain) Only one Main questionnaire by questions all labels plain Girona (Spain) Only one Follow-up by alters end labels plain Ljubljana (Slovenia) 1 and 2 Main questionnaire by questions all labels plain Ljubljana (Slovenia) 1 Follow-up by questions end labels Graphical Ljubljana (Slovenia) 2 Follow-up by alters all labels Graphical Ghent (Belgium) Only one Main questionnaire by questions all labels plain Ghent (Belgium) Only one Follow-up by alters end labels plain

Categorical predictors (country, trait and factor 1 to factor 3)

Multiple classification analysis (MCA) was used

Estimation of the contribution of each factor on reliability and validity.

Intro

71



71

Question order: by alters or by questions.

Intro

72



72

Formulation by alters, with end labels and with a plain lay-out

Kogovšek et al. (2002) “by alters” seems to be more reliable for telephone

interviews.

Intro

73



73

Response category labels: for all categories or for the end points of the

response scale.

Revilla & Saris (2011) find more quality in all labeled.

Költringer (1995) reports no effect of the way of labeling on either reliability or

validity. Personal and telephone interviews and mostly related to attitudinal (i.e.,

not network) variables using vague category labels of the type “rather satisfied”,

“completely agree”

Our study: Category labels are not vague quantifiers. Precise actual frequencies of

behavior and thus additional labels may help respondents give precise answers

about the frequency of contact with their social network.

Intro

74



74

Lay-out of the questions and the web page: plain or graphical display.

Dillman et al. (1998b), who suggest that using a plain questionnaire provided

better results (response rate, less time…) than a graphical display version.

Deutskens et al. (2004) found that visual effects actually increase response quality.

Intro


Meta-analysis results

75

Means of each level of all factors, corrected by the levels of other factors

country trait Girona Ljubl. Ghent trait 1 trait 2 trait 3 trait 4 % of between trait variance .133 .088 .117 .079 .125 .118 .111

% of within trait variance .610 .612 .555 .674 .654 .552 .508

% of between method variance .024 .025 .002 .017 .015 .022 .030

% of within method variance .049 .074 .077 .052 .047 .072 .114

% of between error variance .049 .064 .057 .053 .039 .063 .078

% of within error variance .152 .166 .189 .137 .132 .186 .219

Between reliability .874 .735 .839 .794 .875 .804 .698

Between validity .921 .907 .943 .916 .945 .919 .876

Within reliability .899 .895 .875 .916 .917 .877 .856

Within validity .974 .932 .931 .963 .965 .936 .887

Overall reliability .885 .882 .869 .892 .913 .867 .849

Overall validity .963 .923 .935 .955 .960 .932 .867

% of trait variance at within level .823 .864 .826 .895 .840 .823 .804

Intro



76

factor 1 factor 2 factor 3

by questions by alters all labels end labels plain lay-out

graph. lay-out

% of between trait variance .125 .082 .103 .116 .105 .117

% of within trait variance .650 .508 .595 .600 .593 .608

% of between method variance .013 .031 .006 .042 .026 .008

% of within method variance .035 .120 .070 .074 .082 .045

% of between error variance .065 .046 .061 .054 .059 .058

% of within error variance .136 .215 .181 .147 .163 .183

Between reliability .785 .809 .777 .825 .789 .806

Between validity .948 .880 .961 .856 .902 .958

Within reliability .912 .857 .884 .903 .894 .883

Within validity .975 .888 .942 .932 .927 .965

Overall reliability .895 .852 .873 .893 .884 .868

Overall validity .967 .893 .947 .914 .922 .962

% of trait variance at within level .835 .855 .850 .830 .844 .841

Intro



77

Overall reliabilities and validities are acceptable: around or above 0.85

By traits (pattern is consistent at different levels):

Collaboration (T2)

Advice (T1)

Asking for crucial information (T3)

Socializing (T4)

Overall reliability and validity is higher when:

Social network questions in a survey are organized by questions

All labeled categories

Graphical display questionnaire design (not within validities)

Intro



78

Country: No significant effect on reliability or validity.

% of trait variance at the within level:

Advice � Collaboration � Ask for information � Socializing

Scientific advice tends to be asked rather often to some group

members and not often to other group members by the same ego, but all

egos have a rather similar frequency average, while this occurs to a lesser

extent for socializing.

Intro


Meta-analysis results from Other Studies

79

Meta-Analysis Results from other Studies

Intro



80

Kogovšek , Ferligoj , Coenders , Saris (2002) Coromina, Coenders, Kogovsek (2004) By type of networks: +: Frequency of contact is measured with the highest reliability Feeling of importance of the alter Feeling of closeness between ego and the alter - : Frequency of being upset.

Frequency of contact is less reliably measured over the telephone (speed of

telephone communication) compared with ‘the degree of closeness between the

ego and the alter’ or ‘importance of the alter’)

Intro



81

Answering the question about frequency of contact is probably more

cognitively demanding than answering about the degree of closeness for which the

answer may be more readily available

Cognitively more demanding name interpreter questions would be more reliably

measured by face-to-face than by telephone mode.

Kogovsek & Ferligoj (2005) Reliability Results : Method +reliable: Telephone/by alters (except contact) Personal/by alters -reliable: Telephone/by questions

Intro



82

When Telephone mode only � by alters (higher quality).

Telephone � speed of the mode � smaller elicited networks (would reduce

respondent burden and therefore lead to data of better quality)

BUT mean network size was actually smaller in the face-to-face (6.84) than in the

telephone mode (7.24).

Method: +reliable: face to face +valid: telephone (anonymity of phone) -reliable: telephone -valid: face to face Cognitively more demanding questions would be measured more reliably by face-to-face mode, and that questions that are potentially more sensitive would be measured more reliably in the telephone by alters condition.

Intro



83

It is quite possible that context effects are stronger when the data is collected by questions, since the respondent answers the same question for all alters first and not vice versa.

If we consider a question relating to feelings of closeness, it is more likely that the respondent would compare the alters while answering the question. In that case, it would not really be the actual feelings of closeness towards each individual alter that would be measured, but the feelings of closeness relative to previous alters on the list.

Coromina et al. (2004): Not so clear that telephone method is better than face-to

-face method at the between level.

Telephone produces better quality data at the within level, and an analysis of the

SB such as the one done by Kogovšek et al (2002) is inevitably contaminated by

the within level structure according to E(SB)= ΣW + cΣB

Intro



84

Social support (reliability) Ferligoj & Hlebec (1999)

+: Emotional support (discussion of important matters) +: Informational support (Illness ) Social Companionship (birthday party) -: Instrumental support (material) (importance of the topic to the respondents).

Response scale (the most important predictor of reliability estimates) Hlebec & Ferligoj (2002) +: 5 point category all labels. +: 11 point category scale ending

5 point category scale ending labels Line drawing scale

- : Binary scale

Intro



85

Response scale +reliable: all labelled -reliable: ending labels Name Generator: Recognition data collection enhances the reporting of both strong and weak ties, especially when the measurement scales employed can also measure the strength of ties.

Ferligoj & Hlebec (1999) ( Recognition / Free recall ) no differences Interview design (2nd most powerful predictor for reliability estimates) +reliable : Two measures (when two measures of the same trait are presented in the same interview) -reliable: One measure (a question is presented alone in a questionnaire)

Intro



86

Network size (not significant) 1–5 +valid: 1-5 6+ -valid: 6+ (larger proportion of weak ties and

measurement error more pronounced)

With smaller networks (important alters, consistent answers) the data collection technique (by alters/by questions) is more important than the data collection mode (telephone/face-to-face).

In larger networks: (seems that context effects are no longer so prominent). +valid: telephone -valid: face-to-face

Intro



87

Type of question +reliable: Behavior +valid: behaviour .975 -reliable Emotional, but not very different -valid: emotional .958

Education Up to compl. second. (no effect) College or more

Age +reliable: 40 years or less +valid: 40 years or less -reliable: 41+ -valid: +41 (memory or hearing problems;

more weak ties...)

Intro



88

Gender +reliable: female +valid: male -reliable: male -validity: female (maybe because women have

larger networks and more weak ties and might increase method effect)

Interaction Age / Gender -valid: older women +valid: other categories

Interaction Method / Gender Female/method Male/method +valid: face-to-face (.957) +valid: telephone (.999) - valid: telephone (.930) -valid: face-to-face (.928) Being interviewed about personal relationships by telephone suits both genders better, but face-to-face interviews suit women better than men.

Intro


Project Conclusions

89

Considering egocentered network data as hierarchical, Multilevel recommended.

Qualities at Within, Between and Overall level.

Combination of % from decomposed variance.

Egocentered Network data quality collected by Websurvey:

ordered by Questions

All categories of the scale labeled Overall and Within

Graphical display validity

Documents

Measurement Quality in Social Networks - Pàgines de la UABgrupsderecerca.uab.cat/egolab/sites/grupsderecerca.uab.cat.egolab/files/Coromina... · 1 Universitat de Girona Measurement