Utah School of Computing HCI Validation Richard F. Riesenfeld University of Utah Fall 2004

Utah School of Computing

HCI Validation

Richard F. RiesenfeldUniversity of Utah

Fall 2004

Utah School of ComputingFall 2003 slide slide 22Student Name Server

Major Considerations - 1

• Stage of design - Conceptual, preliminary, detail

• Novelty of project- Do we know what we are doing?

• Number of expected users- How important is this?- How amenable to change will it be?


Major Considerations - 2

• Criticality of the interface - Are lives at stake if there are problems?

• Cost of product- Allocation for testing

• Time available for testing

• Experience of designers and evaluators


Expert Review methods - 1

• Heuristic evaluation- Experts critique it wrt established criteria

• Guidelines review- Does it meet “spec”- Can be an overwhelming list- Bureaucratic approach



• Consistency inspection- Experts check of style, function, form, etc.

• Cognitive walkthrough- Experts perform role of users- Try to assess its success from experience



• Formal usability inspection- Moot court

- Countervailing opinions

- Can be unwieldy


Comparative Evaluations - 1

• Different experts see different issues- Can get caught with conflicting advice*- Limit the number of experts

• Get a “bird’s-eye” view in the beginning- Throw images on a wall, etc.

* “* “For every PhD, there is an equal and opposite PhD”


Comparative Evaluations - 2

• Formal (statistical) methods- Form a hypothesis- Determine dependent variables- Identify independent variables

* “


Usability Labs, etc

• Hard to employ because of time and money constraints in product development- Development cycle schedule- Budgets- Corporate/Cultural Attitude


Controlled Experients

• Statistical testing- Establish level of statistical significance- At the 95% confidence level we know…

• Usability testing- Find flaws in the interface through more

informal (inconclusive) methods- Empirical methods


Human Subjects -1

• Careful, “This isn’t Kansas anymore!”

• Many new dimensions need attention

• Human respect and dignity- Voice generated check outs violated privacy

Military has NO privacy Other cultures treat matters differently


Human Subjects -2

• Real LEGAL issues, so get it right!- Informed consent- Understand your liability- Get it in writing, copy to each party

• Government, or institutional rules- We are not accustomed to this- Need cognizant approvals

IRBs, etc. Research proposals, etc


Observations methods

• Have subjects “think aloud”- Will subjects be honest, etc.

• Use video recording

• Field tests- Study the successes/failures of the interface- Getting access- Reliance on memories

“How is it going?” (We tend to react to most recent)


Destructive Testing

• Hey, can you break this?

• Good for security

• Good for games

• Durability testing appropriate for some environments- ATM in high crime area- Military- Students, they can’t resist a challenge


Competitive Testing -1

• Consumers Union, Road & Track, style- Take several into lab and have a “shoot out”

• Expensive

• Takes skill (like a movie review)- Depends on the criteria- Depends on good and representative judgment


Competitive Testing -2

Major Limitations- Limited coverage of features- Depends on initial user experiences


Surveys

• Tricky business, can lead to nearly any conclusion- Population selection- Question choices- Size- Leading questions, other bias

• Negative bias – users with complaints


Online Surveys

• More issues…


Conclusions

• HCI is a new game

• Not exact science

• Old methods not entirely applicable

• Need newer, faster, light weight, flexible, informal, subjective, intelligent approaches


Recommendations

• Use good judgment

• Trust good judgment- Yours- Others, whom you trust

• Be open to criticism and suggestion


Interpretation

• What is being said?

• What is the real issue?

• What is the real fix?

• RSI is a problem- Pain- Keyboard or mouse?- Need different devices, or speech, or simply a

better mouse and keyboard?


acm.org/~perlman/question.cgi

Please rate the usability of the system. •Try to respond to all the items. •For items that are not applicable, use: NA •Make sure these fields are filled in: System: Email to: •Add a comment about an item by clicking on its icon,

or add comment fields for all items by clicking on Comment All.



•To mail in your results, click on: Mail Data System: Email to: Optionally provide comments and your email address in the box.



1 2 3 4 5 6 7 NA

1. Overall, I am satisfied with how easy it is to use this system strongly disagree strongly agree

2. It was simple to use this system strongly disagree strongly agree

3. I can effectively complete my work using this system strongly disagree strongly agree

4. I am able to complete my work quickly using this system strongly disagree strongly agree

5. I am able to efficiently complete my work using this system strongly disagree strongly agree

6. I feel comfortable using this system strongly disagree strongly agree

List the most negative aspect(s): 3.List the most positive aspect(s): 4.

Top of Form


RHS of Each Row

1 2 3 4 5 6 7 NA

1. strongly disagree

strongly agree



7. It was easy to learn to use this system

8. I believe I became productive quickly using this system

9. The system gives error messages that clearly tell me how to fix problems

10. Whenever I make a mistake using the system, I recover easily and quickly

11. The information (such as online help, on-screen messages, and other documentation) provided with this system is clear

12. It is easy to find the information I needed



13. The information provided for the system is easy to understand

14. The information is effective in helping me complete the tasks and scenarios

15. The organization of information on the system screens is clear

16. The interface of this system is pleasant

17. I like using the interface of this system

18. This system has all the functions and capabilities I expect it to have

19. Overall, I am satisfied with this system



List the most negative aspect(s): 3.List the most positive aspect(s): 4.

Top of Form

Utah School of Computing

END

HCI Validation

Documents

Utah School of Computing HCI Validation Richard F. Riesenfeld University of Utah Fall 2004