Chapter 15: Design and Evaluation in Chapter 15: Design and Evaluation in the Real World: Communicators and the Real World: Communicators and

Advisory SystemsAdvisory Systems

Group 4: Tony Masi, Sam Esswein, Chris Troisi, Group 4: Tony Masi, Sam Esswein, Chris Troisi, Brian RoodBrian Rood

IntroductionIntroduction

Text books make design and usability testing Text books make design and usability testing processes sound straightforward and step-by-step. processes sound straightforward and step-by-step. This is never the case in real world applications. It This is never the case in real world applications. It is only when you become involved in the actual is only when you become involved in the actual design project that the pressure, trade-offs, and design project that the pressure, trade-offs, and demands influence the way the design project is demands influence the way the design project is carried out. In the real world, design and carried out. In the real world, design and evaluation are VERY closely integrated. You don’t evaluation are VERY closely integrated. You don’t do one without the other.do one without the other.

Key ConceptsKey Concepts

Show how design and evaluation are Show how design and evaluation are brought together in development of brought together in development of interactive productsinteractive products

Show how different combinations of design Show how different combinations of design and evaluation methods are used in and evaluation methods are used in practicepractice

Describe the various design trade-offs and Describe the various design trade-offs and decisions made in real-world situationsdecisions made in real-world situations

Key IssuesKey Issues

The number of iterations through the design-The number of iterations through the design-evaluate cycle depend on the requirements of the evaluate cycle depend on the requirements of the projectproject

Many practical issues and unexpected events Many practical issues and unexpected events must be dealt with by the design teammust be dealt with by the design team

No two projects are ever exactly the same; each No two projects are ever exactly the same; each will face a different set of constraints, demands, will face a different set of constraints, demands, and crisesand crises

Real World Usability CasesReal World Usability Cases

http://www.asktog.com/columns/042ButterflyBallot.http://www.asktog.com/columns/042ButterflyBallot.htmlhtml

- Florida Butterfly Ballot - Florida Butterfly Ballot

http://www.digitwireless.com/http://www.digitwireless.com/ - FasTap Keypad - FasTap Keypad

http://www.consult-me.co.uk/csc-case-studies.htmhttp://www.consult-me.co.uk/csc-case-studies.htm - - Some other usability case studiesSome other usability case studies

Designing Mobile CommunicatorsDesigning Mobile Communicators

NokiaNokia• Communicator for Communicator for

adultsadults

PhilipsPhilips• Communicator for Communicator for

childrenchildren

BackgroundBackground

Mobile communicators Mobile communicators are devices that are devices that commonly push the commonly push the limits of technology. limits of technology. They combine many They combine many functions such as functions such as those listed to the those listed to the right into one small right into one small device. Therefore, a device. Therefore, a key challenge is how key challenge is how to make these devices to make these devices useable and useable and affordable to most affordable to most people. people.

Send and receive Send and receive telephone calls, email, telephone calls, email, faxes, and other faxes, and other messagesmessages

Keep contact info, Keep contact info, journal entries, journal entries, calendars, and other calendars, and other notesnotes

Watch stock and other Watch stock and other finance reportsfinance reports

BackgroundBackground

Other considerations:Other considerations: Device should be small and lightweightDevice should be small and lightweight

• Made of light materialsMade of light materials• Small enough to fit in pocket or small bagSmall enough to fit in pocket or small bag• Software must work with limited screen size and memorySoftware must work with limited screen size and memory

What will user be doing while using the device?What will user be doing while using the device?• Device should work in all conditionsDevice should work in all conditions

Various noise levelsVarious noise levels Various lightingsVarious lightings

• Device should take little effort to operate so user can Device should take little effort to operate so user can concentrate on other things at the same time concentrate on other things at the same time

like driving?like driving?• Hands free mode?Hands free mode?• Simple operations should be one-handedSimple operations should be one-handed

Answer callAnswer call Browse internetBrowse internet

BackgroundBackground

Other considerations:Other considerations: Device should account for distractions that may occurDevice should account for distractions that may occur

• Interaction sequences should be easy to return to and continue Interaction sequences should be easy to return to and continue after interruptionafter interruption

• Internet trade off: How long should the device remain Internet trade off: How long should the device remain connected?connected?

Tasks tend to be time-critical, triggered by other people or Tasks tend to be time-critical, triggered by other people or events, relatively brief, low in terms of attention to be events, relatively brief, low in terms of attention to be applied to the task, and very personalapplied to the task, and very personal• Flow among tasks should be smoothFlow among tasks should be smooth• Easy flow between related functions such as contact list and Easy flow between related functions such as contact list and

telephone functiontelephone function Must be simple and not involve much training (if any)Must be simple and not involve much training (if any) Needs to be robust and reliableNeeds to be robust and reliable

Designing Nokia’s Mobile Designing Nokia’s Mobile CommunicatorCommunicator

What kind of lifecycle?What kind of lifecycle?• Iterative user-centered approachIterative user-centered approach• Top level design concept cycle (Fig 15.2)Top level design concept cycle (Fig 15.2)

Which methods to use?Which methods to use?• Ethnographic researchEthnographic research• Scenarios and task models (Fig 15.3, 15.4)Scenarios and task models (Fig 15.3, 15.4)

Confidential product issues:Confidential product issues:• First in the market is keyFirst in the market is key• Evaluation must be very limitedEvaluation must be very limited• No real usersNo real users

Physical aspects:Physical aspects:• Screen sizeScreen size• Number of buttons versus functionalityNumber of buttons versus functionality

Soft keys with changing functions or valuesSoft keys with changing functions or values Hard coded keys that always return same valueHard coded keys that always return same value

Consistency issuesConsistency issues• Internal consistency (within mobile software)Internal consistency (within mobile software)• External consistency (with desktop software) External consistency (with desktop software)

User testingUser testing• None before release (confidentiality)None before release (confidentiality)• Summative testing & questionnaires afterSummative testing & questionnaires after

Designing Nokia’s Mobile Designing Nokia’s Mobile CommunicatorCommunicator

Designing Philips’ communicator Designing Philips’ communicator for childrenfor children

design cycle:design cycle:iterative and evolutionaryiterative and evolutionary

which methods:which methods:low-fidelity prototyping low-fidelity prototyping participatory designparticipatory designinterface metaphorsinterface metaphors

physical aspects:physical aspects:color, shape, size, robustnesscolor, shape, size, robustnesspen input pen input bags to protect screenbags to protect screen

Designing Philips’ communicator Designing Philips’ communicator for childrenfor children

user involvement:user involvement:children involved throughoutchildren involved throughoutprototypes evaluated constantlyprototypes evaluated constantlyinvaluable insights for the designers invaluable insights for the designers

lessons learned:lessons learned:agree on assumptions in requirementsagree on assumptions in requirementsthink of follow-on projects early onthink of follow-on projects early onusers are not designersusers are not designersact quick and dirty if necessaryact quick and dirty if necessary

Case study 2: Case study 2: A telephone response information system A telephone response information system

(TRIS)(TRIS) Interactive voice response systems are common in Interactive voice response systems are common in

government offices and large companies. Do you government offices and large companies. Do you know of examples that you have used?know of examples that you have used?

Why are these systems often so frustrating to use?Why are these systems often so frustrating to use?Forming a mental model is difficult because there is Forming a mental model is difficult because there is no visual feedback and the user must remember the no visual feedback and the user must remember the menu structuremenu structure

Many menus and deep menus are particularly Many menus and deep menus are particularly difficult difficult

Why was TRIS difficult to use?Why was TRIS difficult to use? Having to remember the menu structure.Having to remember the menu structure. The programmers traded computational elegance The programmers traded computational elegance

for usability, e.g., the system asked for social for usability, e.g., the system asked for social security number and employee identification security number and employee identification number, confusing users who did not have both.number, confusing users who did not have both.

TRIS was comprised of different systems each with TRIS was comprised of different systems each with its own interaction style. Users were not told this its own interaction style. Users were not told this but when they moved between the systems they but when they moved between the systems they experienced sudden, unexplained changes. experienced sudden, unexplained changes.

How was TRIS evaluated?How was TRIS evaluated?

A combination of techniques were used:A combination of techniques were used:- a review of the literature provided information - a review of the literature provided information about problems with interactive voice response about problems with interactive voice response systemssystems- expert reviews - expert reviews - GOMS analysis of the proposed redesign- GOMS analysis of the proposed redesign

The redesign was implementedThe redesign was implemented- usability tests confirmed that the redesigned - usability tests confirmed that the redesigned system offered better usability than the original system offered better usability than the original design design

Why was using different methods Why was using different methods valuable?valuable?

The evaluators were able to build-up a broad The evaluators were able to build-up a broad picture of usability problems.picture of usability problems.

Using GOMS and heuristic evaluation they could Using GOMS and heuristic evaluation they could explore the potential benefits of the redesigned explore the potential benefits of the redesigned system.system.

User testing enabled them to confirm that the User testing enabled them to confirm that the redesigned system offered better usability.redesigned system offered better usability.

User satisfaction questionnaires confirmed that User satisfaction questionnaires confirmed that users preferred the redesigned system.users preferred the redesigned system.

Key pointsKey points

Design involves trade-offsDesign involves trade-offs Design space for making changes when Design space for making changes when

upgrading a product is limitedupgrading a product is limited Cycles of rapid prototyping and evaluation Cycles of rapid prototyping and evaluation

allow designers to examine alternativesallow designers to examine alternatives Simulations are useful when evaluating Simulations are useful when evaluating

systems used by large numbers of peoplesystems used by large numbers of people Piecing together evidence from a variety of Piecing together evidence from a variety of

sources can be valuablesources can be valuable