Scale perception bias in Likert scale questions: Improved validity through anchoring vignettes?

Scale perception bias in Likert scale questions: Improved validity through anchoring vignettes?

Klaus Gebel

Centre for Chronic Disease Prevention

James Cook University

German tourists in SpainGerman tourists in Spain

Political efficacy in China and MexicoPolitical efficacy in China and Mexico

Health inequalities between Indigenous and Health inequalities between Indigenous and non-Indigenous Australiansnon-Indigenous Australians

Anchoring vignettesAnchoring vignettesExample: Self-rated healthExample: Self-rated health

ExcellentExcellent Very goodVery good GoodGood FairFair PoorPoor

Most common measure of health Often negative association with actual health

SF-36: In general, would you say your health is:

Anchoring vignettesDifferential Item Functioning (DIF)

SelfSelf11

Low Low

HighHigh

Jo1

Pat1

Chris1

SelfSelf22

Jo2

Pat2

Chris2

Low

High

Chris2

Jo2

Pat2

SelfSelf22

Recoding depending on self-rating and rating of vignettes

Self & vignettes Recoding

x>V1>V2>V3 7x=V1>V2>V3 6V1>x>V2>V3 5V1>x=V2>V3 4V1>V2>x>V3 3V1>V2>x=V3 2V1>V2>V3>x 1x>V1>V2=V3 7x>V1=V2=V3 7

Next steps

Conversion from 7-point scale back to 5-point scale

Paired t-test to compare DIF-adjusted vs unadjusted scores

Wald’s test to compare associations of DIF-adjusted vs unadjusted scores with continuous or binary outcome measures

Design of vignettes

Ideally equally spaced through distribution of self-assessments

Avoid extremes Gender specific Ideally 2-3 vignettes Design vignettes in focus group Test them with sub sample

Example: Safety from traffic and cycling / walking

Safety and accessibility are the two most important environmental factors affecting activity participation across the lifespan.

DiPietro 2012, Human Kinetics

“”

Background

Background

Safety from traffic

Speed of traffic

Volume of traffic

Separation from traffic

Results

Very unsafe

A little unsafe

Neither safe nor unsafe

A little safe

Very safe

How safe do you think it is to ride a bicycle in your local area?

1 2 3 4 5

Predictive validity of perceived safety from traffic

Methods

Baseline data collected 2013Cycling and walking measured with travel diary

Outcome measures

• Cycle to work/study y/n (Probit)

• Frequency of cycling (ordered Probit)

• Mins/week of cycling and walking (Poisson)

Results

n=871 58% women 18-55 (37±11.1 years) 14% use bicycle as main way travel mode to

work/study 29% cycle at least 1-2 days/week

Results

Perceived safety from traffic

Results

Ratings of vignettes

0

100

200

300

400

500

600

Very unsafe A littleunsafe

Neithersafe norunsafe

A little safe Very safe0

50

100

150

200

250

300

350

Very unsafe A little unsafe Neither safenor unsafe

A little safe Very safe0

100

200

300

400

500

600

Very unsafe A little unsafe Neither safenor unsafe

A little safe Very safe

Results

DIF adjusted perceived safety from traffic

Results

Unadjusted Adjusted for DIF

Wald’s test p-value

Cycle to work/study y/n 0.63 0.79 0.03

Frequency of cycling 0.71 0.78 0.06

Mins/week of cycling 0.53 0.78 >0.01

Mins/week of walking 0.48 0.62 0.01

Predictive validity of perceived safety from traffic

Conclusion

Significant scale perception bias Anchoring vignettes predictive validity Anchoring vignettes might be a powerful tool

to improve validity of Likert-scale items

Opportunistic evaluation of new bicycle paths in Cairns

Bike path evaluation in CairnsBike path evaluation in Cairns

Thank you for your attention!Thank you for your attention!

Questions?Questions?

klaus.gebel@jcu.edu.au klaus.gebel@jcu.edu.au