Usability Testing Methods: Subjective Measures Part I and II by Joe Dumas*

Summary presented by Meeyong RaoHCDE 517 -Usability Testing Winter 2014

*From: The Common Ground, The Newsletter of Usability Professionals (1998)

Part I – Creating Effective Questions and Answers

Three Sources of Distortion in Surveys

1. Participants• Biases: yea-saying, central tendency, leniency, social

desirability

Three Sources of Distortion in Surveys Cont.

2. Interviewer• Training

• Experience

3. Of the task situation itself• Method of administration: Self-administered vs. F2F

• Effect of question wording

• Memory limitation: forgetting & telescoping

Three Sources of Distortion in Surveys Cont.

The Choice of Answer Options

• Questions about frequency of behavior are influenced by the distribution of pre-coded answer options

• Example: if a participant is asked about how frequently s/he watches TV…

A low range from ½ hour to 2 and ½ hours reported lower averages of watching TV than a higher range including 2 and ½ hours to 4 and ½ hours

Part II – Measuring Attitudes and Opinions

Most Used Question Format for Opinions of Usability: Likert Scale

Q. Common Ground is a quality newsletter.

Strongly Agree Strongly Disagree

1 2 3 4 5

• #’s represent equal intervals and how much one is preferred over another

• 5, 7, or 9-point scale (provides true midpoint); little value in having more than 9 or 10

• If measuring ONLY “the intensity of conviction” then do not include neutral point on scale

Scale End Point Labels

Very Difficult Very Easy

1 2 3 4 5

• Scale doesn’t measure one single dimension

• Measures two dimensions: difficulty and easy

• Saying a product is “difficult” to use is not the same as saying it is “not at all easy”

• These labels activate two different cognitive structures; should activate only one

Frequently Used Format for Opinions of Usability: Semantic Differential

Favorable - - - - - - UnfavorableUnattractive - - - - - - AttractivePleasant - - - - - - Unpleasant

• Respondents check one of the 7 lines between word pairs• Polar opposites• Connotative meaning, Not denotative meaning

Ex. School mascot - Husky• Denotative: animal• Connotative: strength/power

Negative Numbers in Scale

Q. Please rate how successful you have been in life.

Not at Successful Very Successful

-3 -2 -1 0 1 2 3

• Participants give significantly more positive ratings using negative ratings scale compared to positive scale of 1 to 7

Scales of Measurement: Nominal

Q. What is your gender?

1. Male

2. Female

• Numbers assigned to answer options do not have meaning except for distinction

• Numbers don’t measure anything. Could be letters of the alphabet instead.

Scales of Measurement: OrdinalQ. When you are considering purchasing a smart phone, please rank the importance of each of the following on your decision. Please use a ranking from 1 to 5, where 1 is most important and 5 is least important.

___ Apps___ Battery life___ Quality of camera___ Operating system___ Voice control

• Numbers represent ordering (or magnitude) of a property of an object

• Doesn’t indicate size of the difference

Scales of Measurement: Interval

• Numbers represent equal intervals

• Shows how much one is preferred over another

Examples: Likert scale and semantic differential

Scales of Measurement: Ratio

Q. How many hours per day do you use an iPhone?

___ Hours

• Similar to interval scales except has an absolute zero

Thoughts for Usability Practitioners Using Questions as Subjective Measures

1. Always consider why you are asking a question and how participants might interpret the words/format

2. When you find a question that works, stick with it

3. Use questions to stimulate discussions

4. Never restrict your usability evaluation to one measure, especially a subjective measure