Www.site.uottawa.ca/~elsaddik 1 Unit B-Introduction (c) elsaddik SEG 3210 User Interface Design & Implementation Prof. Dr.-Ing. Abdulmotaleb El Saddik

ww

w.s

ite.

uo

ttaw

a.ca

/~el

sad

dik

1Unit B-Introduction(c) elsaddik

SEG 3210

User Interface Design & Implementation

Prof. Dr.-Ing. Abdulmotaleb El Saddik

University of Ottawa (SITE 5-037)

(613) 562-5800 x 6277

elsaddik @ site.uottawa.ca

abed @ mcrlab.uottawa.ca

http://www.site.uottawa.ca/~elsaddik/

ww

w.s

ite.

uo

ttaw

a.ca

/~el

sad

dik


Unit B: UI Evaluation

1. Objectives of User Interface Evaluation

2. Where Evaluation Fits in the Development Process

3. A Preliminary Case Study: Hotel Reservations

4. Overview of Interface Evaluation Methods

5. Details: Heuristic Evaluation

6. Malfunction Analysis

7. Details: Videotaped Evaluation

8. Details: Experiments

9. Details: Usability Engineering

10. Details: Cognitive Walkthroughs

11. Key Points to Review

ww

w.s

ite.

uo

ttaw

a.ca

/~el

sad

dik



Key objective of both UI design and evaluation:

•Minimize malfunctions

Key reason for focusing on evaluation:• Without it, the designer would be working “blindfold”

• Designers wouldn’t really know whether they are solving customer’s problems in the most productive way

ww

w.s

ite.

uo

ttaw

a.ca

/~el

sad

dik



Questions answered by various evaluation techniques:

1. What is the user’s real task?• Prevent later malfunctions

• by doing evaluation as part of requirements analysis

• Present and work with a UI• to help formulate the requirements

• Inappropriate tasks/requirements are a major source of malfunctions2. What problems do or might users experience with the UI?

• Directly find malfunctions3. Which of several alternative UI’s is better?

• Pick the version that leads to fewer malfunctions4. Has the UI met usability targets?

• Ensure that malfunction counts are sufficiently low5. Does the UI conform to standards?

• Leverage of collective wisdom to reduce malfunctions

ww

w.s

ite.

uo

ttaw

a.ca

/~el

sad

dik



But, in order for evaluation to give feedback to designers...

...we must understand why a malfunction occurs

Malfunction analysis:• Determine why a malfunction occurs• Determine how to eliminate malfunctions

We will discuss this while working through this unit.

ww

w.s

ite.

uo

ttaw

a.ca

/~el

sad

dik


2. Where Evaluation Fits in the Development Process

Throughout the lifecycle!• During rough sketching or prototyping• During iterative design

The more evaluation the better• Especially when users are involved

Formative evaluation:• When designing and maintaining software that we are

developing

Summative evaluation:• When judging a finished product developed by someone else

ww

w.s

ite.

uo

ttaw

a.ca

/~el

sad

dik



UI Evaluation performed for Forte Travelodge Performed in a special usability lab

Aims:• Identify and eliminate malfunctions

• Hence make system easier to use• Avoid business difficulties caused by these malfunctions• Develop improved training material and documentation

• Avert potential malfunctions by teaching users how to avoid them

Setup of IBM usability lab:• Resembles TV studio

• Microphones and video equipment

• One way mirror

• Technicians, observers sit on one side

• Users sit on other side in realistic environment• User environment resembles reception desk

• Non-threatening

ww

w.s

ite.

uo

ttaw

a.ca

/~el

sad

dik



Aspects of system to be evaluated:• How quickly can a booking be made?

• (while operator is on telephone)• Is each screen productive to use?• Are help and error messages effective?• Can non-computer-literate operators use the system?

• Is complexity minimized?• Is training and documentation effective?

ww

w.s

ite.

uo

ttaw

a.ca

/~el

sad

dik



Procedure:• 15 common task scenarios developed:

• Among others: basic registration, cancellation, request for specific room, extension of existing stay etc.

• Four days of testing with multiple users performing various sets of tasks

• Users were told evaluation is of system, not them

• All actions were recorded

• Debriefing sessions held• Videos then analyzed for malfunctions

• 62 identified

• Priorities:• Navigation speed needs improvement• Screen titles and formats need tuning• Hard to refer to documentation• Physical difficulties with telephone headsets and furniture

ww

w.s

ite.

uo

ttaw

a.ca

/~el

sad

dik



Results:• Higher productivity of booking staff

• tasks completed more quickly

• guest requirements better met• Training costs kept low• Morale kept high

• More customers booked by phone

• 14500 27000 per week

ww

w.s

ite.

uo

ttaw

a.ca

/~el

sad

dik


4. Overview of Interface Evaluation Methods

• Three types of methods• Passive evaluation

• Active evaluation

• Predictive evaluation / usability inspections • All types of methods useful for optimal results

• Used in parallel

• All attempt to prevent malfunctions

• Before trying methods, do pilot studies first

ww

w.s

ite.

uo

ttaw

a.ca

/~el

sad

dik


4. Passive evaluation

• Usage of software is monitored• Performed while prototyping, in alpha test and later• Does not actively seek malfunctions

• only finds them when they happen to occur infrequent (but possibly severe) malfunctions may not be found

• Generally requires realistic use of a system• Users become frustrated with malfunctions

Gathering Information:a) Problem report monitoring:

• Users should have an easy way to register their frustration / suggestions

• Best if integrated with software

ww

w.s

ite.

uo

ttaw

a.ca

/~el

sad

dik


4. Passive evaluation: Gathering Information

b) Automatic software logs• Can gather much data about usage

• command frequency• error frequency and pre-error patterns• undone operations (a sign of malfunctions)

• Privacy is a concern

• System must be designed for testability (DFT)

• Logs can be taken of:• just keystrokes, mouse clicks• full details of interaction

• The latter make accurate playback easier

ww

w.s

ite.

uo

ttaw

a.ca

/~el

sad

dik


4. Passive evaluation: Gathering Information

c) Questionnaires / surveys• Useful to obtain statistical data from large numbers of users

• Proper statistical means are needed to analyze results

• Gathers subjective data about importance of malfunction• automated logs omit importance• less frequent malfunctions may be more important• users can prioritize needed improvements

• Limit on number of questions

• Very hard to phrase questions well

• Questions can be closed- or open-ended

ww

w.s

ite.

uo

ttaw

a.ca

/~el

sad

dik


4. Active evaluation

• Actively study specific activities performed by users• Performed when prototyping and later

Gathering Information:d) Experiments & usability engineering

• Prove hypotheses about measurable attributes of one or more UI’s• e.g. speed/learning/accuracy/frustration

• In usability engineering test against preset targets

• Can be expensive

• Knowledge of statistics needed

• Hard to control for all variablese) Observation sessions

• Also called ‘interpretive evaluation’

• Simple observation or cooperative evaluation

• Described in detail later

ww

w.s

ite.

uo

ttaw

a.ca

/~el

sad

dik


4. Predictive evaluation

• Studies of system by experts rather than users• Performed when UI is specified and later

• useful even before prototype developed• Can eliminate many malfunctions before users ever see

software• Also called ‘usability inspections’

Gathering Information:f) Heuristic evaluation

• Based on a UI design principle document

• Analyze whether each guideline is adhered to in the context of the task and users

• Can also look at adherence to standardsg) Cognitive walkthroughs

• Step-by-step analysis of:• steps in task being performed• goals users form to perform these tasks• how system leads user through tasks

ww

w.s

ite.

uo

ttaw

a.ca

/~el

sad

dik


Summary of evaluation techniques

Technique When to use

a) Problem reporting Always

b) Automatic logs In any moderately complex system and whenever there are large numbers and commands

c) Questionnaires Whenever there are large number of users

d) Experiments & Usability Engineering

In special cases where it is hard to choose between alternatives, or when fine tuning

e) Observation sessions Almost always, especially when user has to interact with a client while using the system

f) Heuristic evaluation Always

g) Cognitive Walkthrough When usability must be optimized

ww

w.s

ite.

uo

ttaw

a.ca

/~el

sad

dik


Comparison of key questions to evaluation techniques

A Prob

B Log

C Q?

D Exp

E Obs

F Heu

G Wlk

What is task? ++ ++ +

What are current malfunctions?

+ + ++ ++ ++ ++ ++

Which UI is better? ++ ++ + +

Has UI met targets? + + + ++ + 0

Does UI conform to standards?

+ ++

++: Very good techniques

+ : OK technique

0 : possible technique

ww

w.s

ite.

uo

ttaw

a.ca

/~el

sad

dik



• A type of predictive evaluation• Use HCI experts as reviewers instead of users

• Benefits of predictive evaluation:• The experts know what problems to look for

• Can be done before system is built

• Experts give prescriptive feedback• Important points about predictive evaluation:

• Reviewers should be independent of designers

• Reviewers should have experience in both the application domain and HCI

• Include several experts to avoid bias

• Experts must know classes of users

• Beware: Novices can do some very bizarre things that experts may not anticipate

ww

w.s

ite.

uo

ttaw

a.ca

/~el

sad

dik



• Planning for heuristic evaluation• Based on UI guidelines (heuristics about what is best)• Multiple passes needed

• one pass to look for each kind of problem• passes to follow different routes through screens and dialogues (i.e. different

tasks)

• 1-2 hour sessions is good• 1 expert evaluator finds only 33% of problems• 5 evaluators needed to find 75% of problems• 15 more to find about 99%

• Example heuristics for heuristic evaluation (Many more to be covered later in course)

• Use simple and natural dialogue• Speak the user’s language• Minimize memory load• Be consistent• Provide feedback• Provide clearly marked exits• Provide shortcuts• Provide good error messages• Prevent errors

ww

w.s

ite.

uo

ttaw

a.ca

/~el

sad

dik


6. Malfunction Analysis

A disciplined approach to analyzing malfunctions

• Provides feedback into the redesign process

1. Play protocol, searching for malfunctions

2. Answer four distinct questions:• Q1. How is the malfunction manifested?

• What do you notice and who noticed it?• Q2. At what stage in the interaction is it occurring?

• Goal forming, action decision, action execution, interpretation of results

• Q3. At what level of the user interface is it occurring?• Physical element level to task level

• Q4. Why is it occurring?• What is its root cause

3. List and prioritize possible cures

ww

w.s

ite.

uo

ttaw

a.ca

/~el

sad

dik


Q1. How is the malfunction manifested?

a) Malfunctions detected by the system (easiest to detect)

• omission of an argument• incorrect date format

Cure: • Better prompts, consistency, visible examples, more forgiving

of alternatives

b) Malfunctions detected by the user during operation

• taking wrong path in menu hierarchy• not finding required help• not being able to perform a certain action• not being able to tell which state system is in

Cure:• Improve functionality, feedback, clarity, simplicity

ww

w.s

ite.

uo

ttaw

a.ca

/~el

sad

dik


Q1. How is the malfunction manifested?

c) Malfunctions undetected (until later)• output produced is wrong due to wrong inputs• unnecessary work performed

Cure: • Improve feedback indicating consequences of input; simplify

d) Inefficiencies• excessive response time• excessive think time• unnecessarily long command sequences• unnecessary repetitions• complex operations that require use of reference

Cure: • Simplify, speed system up

ww

w.s

ite.

uo

ttaw

a.ca

/~el

sad

dik


Q2. What Stage in the Interaction the Malfunction Occur?

a) When the user decides on next goal (Forms an intent to do inappropriate thing)• decides to empty a field because user thinks it is unimportant

(when it is important)• decides to charge default exchange rate (when should obtain

current exchange rate)

Cure: • Lead user through task better; better feedback; better training

b) When the user specifies the action (Action does not match the goal)• deletes the record instead of emptying a field• charge reciprocal of exchange rate

Cure:• Improve clarity, feedback, prompts, conceptual model

ww

w.s

ite.

uo

ttaw

a.ca

/~el

sad

dik


Q2. What Stage in the Interaction the Malfunction Occur?

c) When the system executes the action• Defects in functionality

Cure:• Fix functionality in normal way

d) When the user interprets the resulting system state• thinks bank account has been debited when it has not• thinks system has ‘hung’ when it has not• thinks some data must be entered when it is the default• cannot understand resulting error message

Cure:• Better feedback, better conceptual model

ww

w.s

ite.

uo

ttaw

a.ca

/~el

sad

dik


Q3. At Which Level Does the Malfunction Occur?

a) Task level (Task and goals not supported)• What the user wants to do cannot be done by the system• Functionality is not provided

Cure: • Add functionality

b) Conceptual level (User has wrong mental model; does not understand intended conceptual model)• thinks that money is being deducted from bank account when it

is being charged to a credit card• thinks that dragging a file to the desktop means they are no

longer on the disk• thinks that dragging a disk to the trash can icon deletes disk

contents

Cure: • make conceptual model clearer; improve metaphors

ww

w.s

ite.

uo

ttaw

a.ca

/~el

sad

dik



c) Interaction style level (system wide problem)• does not know how to pull down a menu• scrolls a page instead of a line• goes to next screen instead of scrolling• retypes command after an error instead of editing it

Cure: • make operation of the interface more intuitive and consistent

d) Interaction element level (specific detail inappropriate)• selects wrong button because label is misinterpreted• specifies invalid command syntax• specifies wrong code for option

Cure: • More attention to details of the interface, simplification

ww

w.s

ite.

uo

ttaw

a.ca

/~el

sad

dik



e) Physical element level (Physical execution incorrect)• presses wrong key accidentally• clicks on wrong pixel in image• out-types machine (actions lost)• types ahead when system is computing; keystrokes later

applied to wrong action

Cure: • Defenses to protect user from consequences; better hardware

design; fix bugs in code

ww

w.s

ite.

uo

ttaw

a.ca

/~el

sad

dik


Q4. Why Does the Malfunction Occur?

Lack of (on the part of the user):• Motivation:

• Poor job satisfaction• Attention:

• User is pre-occupied with other things.• Input information processing:

• No feedback provided to tell user what is going on

• or cues provided by the system are not recognized

• or cues are misinterpretedCures: Clearer, more consistent feedback

• Discrimination: • user is unable to tell certain things apart

• e.g. red/green colour discrimination

• e.g. two icons that are similarCures: Improved expression of information

ww

w.s

ite.

uo

ttaw

a.ca

/~el

sad

dik



• Physical coordination:• e.g. wrong item selected because of difficulty positioning cursor

with mouse.Cures: Alternate interaction mechanisms, better

feedback• Recall:

• User did not remember command , syntax etc.Cures: Better mnemonics, online help, quick lookup

mechanisms, command completion• Knowledge / lack of learning:

• User does not have business or software knowledge to make right choice.

ww

w.s

ite.

uo

ttaw

a.ca

/~el

sad

dik



Learning difficulties that cause malfunctions: • Learning is difficult

• users get frustrated

• learning takes time; can be hard to apply• Learners make ad-hoc interpretations

• they may not recognize their problem

• they may falsely think they have a problem• Learners generalize from what they know

• they assume computers work like manual methods

• they assume consistency• Learners have trouble following directions

• they often ignore them even if they see them

• they do not easily understand them

ww

w.s

ite.

uo

ttaw

a.ca

/~el

sad

dik



• Problems and features interact• they do not see that one problem can cause another

• Prerequisites and side-effects confuse learners• Help facilities do not always help

• they do not know what to ask for

• too much detail is often provided

• Other causes of malfunctions:• Excessive resource demands

• External events (e.g. noise)

• Misleading or inadequate training

• Unrealistic task definitions

• Intrinsic human variability

ww

w.s

ite.

uo

ttaw

a.ca

/~el

sad

dik



A software engineer studies users who are actively using the user interface• To observe what problems they have

• Rather than to measure numbers• The sessions are videotaped• Can be done in user’s environment

Activities of the user:• Performs pre-defined tasks

• With or without detailed instructions on how to perform them• Preferably talks to herself as if alone in a room

• Yields ‘think-aloud protocol’

• This process is called ‘co-operative’ evaluation when the software engineering and user talk to each other

ww

w.s

ite.

uo

ttaw

a.ca

/~el

sad

dik



The importance of video:• Without it, ‘you see what you want to see’

• You interpret what you see based on your mental model• In the ‘heat of the moment’ you miss many things• Minor details (e.g. body language) captured• You can repeatedly analyze, looking for different problems

Tips for using video:• Several cameras are useful• Software is available to help analyse video by dividing into

segments and labeling the segments• Evaluation can be time consuming so plan it carefully

ww

w.s

ite.

uo

ttaw

a.ca

/~el

sad

dik


Steps for videotaped evaluation

1. Select 6 to 8 representative users per user class• E.g. client, salesperson, manager, accounts receivable

2. Invite them to individual sessions• Sessions should last 30-90 minutes

• Schedule 4-6 per day3. If system involves user's clients in the interaction:

1. Have users bring important clients

2. or have staff pretend to be clients4. Select facilitators/observers and notetakers5. Prepare tasks:

• Select the most commonly used tasks plus a few less important tasks

• Write task instructions for users

• Estimate the time it will take to complete each task plus extra time for discussion

6. Prepare notebook or form for organizing notes

ww

w.s

ite.

uo

ttaw

a.ca

/~el

sad

dik



7. Set up and test equipment1. Hardware on which to run system

2. Audio or video recorder

3. Software logs8. Do a dry run (pilot study)!9. At the Start of an Observation Session

• explain:• nature of project• anticipated user contributions• why user's views are important• focus is on evaluating the user interface, not evaluating the user• all notes, logs, etc., are confidential• user can withdraw at any time• usage of devices• relax!

• Sign informed consent form: • very important

ww

w.s

ite.

uo

ttaw

a.ca

/~el

sad

dik



10. Start user verbalizing as they perform each task (thinking aloud)• For co-operative evaluation, software engineer also verbalizes

• Appropriate questions to be posed by the observing software engineer:

Question Malfunction if What do you want to do? They do not know; the system

cannot do what they want

What do you think would happen if ...?

They do not know; they give wrong answer.

What do you think the system has done?


What do you think is this information telling you?


Why did the system do that? They do not know; they give wrong answer.

What were you expecting to happen?

They had no expectation; they were expecting something else.

ww

w.s

ite.

uo

ttaw

a.ca

/~el

sad

dik



11. Hold a wrap-up interview (de-briefing)1. What were the most significant problems?

2. What was most difficult to learn?

3. Etc.12. Analyze the videotape to find malfunctions

Lab exercise: • Videotaped evaluation of a software product

ww

w.s

ite.

uo

ttaw

a.ca

/~el

sad

dik


8. Details: Experiments (Details in Experimentation)

1. Pick a set of subjects (users)1. A good mix to avoid biases

2. A sufficient number to get statistical significance (avoid random happenings effect results)

2. Pick variables to test• Independent: Manipulated to produce different conditions

• Should not have too many• They should not affect each other too much• Make sure there are no hidden variables

• Dependent: Measured value affected by independent3. Develop a hypothesis

• A prediction of the outcome

• Aim of experiment is to show this is correct

• E.g. Some change in an independent variable causes some change in a dependent variable

ww

w.s

ite.

uo

ttaw

a.ca

/~el

sad

dik


8. Details: Experiments (Details in Experimentation)

4. Design experiments to test hypotheses• Create a null (inverse) hypothesis

• e. a change in independent variable causes no change in dependent variable

• Disprove null hypothesis!

• Experiment design is difficult.5. Conduct experiments6. Statistically analyze results to draw conclusions

• e.g. using ‘t-tests’

• conclusions will be correct within a margin of error 19 times out of 20

7. Decide what action to take based on conclusions

ww

w.s

ite.

uo

ttaw

a.ca

/~el

sad

dik


Two major types of UI experimentation

• Traditional• Micro level

• (e.g. testing which colour is best for a certain icon)• Only done when usage will be very high or you are a university

researcher!

• Usability engineering• Tests significant part of system• Relaxes ‘scientific constraints’

• Because we cannot control all variables• Used to prove hypotheses that certain usability goals have

been met• More later

You should understand experimentation• Much UI research is experimental• You have to interpret and apply others’ results

ww

w.s

ite.

uo

ttaw

a.ca

/~el

sad

dik


Example Experiment: Text Selection Schemes

Early GUI research at Xerox on the Star Workstation

• Traditional experiments• Results were used to develop Macintosh

Goal of study:• Evaluate how to select text using the mouse

Steps:

1. Subjects• Six groups of four

• In each group, only two are experienced in mouse usage

ww

w.s

ite.

uo

ttaw

a.ca

/~el

sad

dik



2. Variables• Independent:

• Selection schemes:• 6 strategically chosen patterns involving• --> Which mouse button (if any) could be double/triple/quad clicked

to select character/word/sentence• --> Which mouse button could be dragged through text• --> Which mouse button could adjust the start/end of a selection

• Dependent• Selection time

• Selection errors

3. Hypothesis• Some scheme is better than all others

ww

w.s

ite.

uo

ttaw

a.ca

/~el

sad

dik



4. Detailed experiment design• Null hypothesis: No difference in schemes• Assign a selection scheme to each group• Train the group in their scheme• Measure task time and errors

• Each subject repeated 6 times

• A total of 24 tests per scheme

5. Conduct Experiment

6. Analysis• F-test used - scheme F found to be significantly better

• Point and draw through with left mouse

• Adjust with middle mouse

7. Action• Try another combination similar to scheme F• Left mouse can be double-clicked

ww

w.s

ite.

uo

ttaw

a.ca

/~el

sad

dik


Questions to ask when reviewing experiments

Not all published experiments are done well!• Were users adequately prepared?• Were tasks complex enough to allow adequate evaluation?• Did the task become boring to the users?• Although effects are found to be statistically significant, does

that matter?• Maybe not if a particular task is rarely performed

• Are there any other possible interpretations?• Maybe users have learned to do better at task B because they did

task A first!• Are dependent variables consistent?

• e.g. users may prefer slower method• Can results be generalized?

• Maybe selection results also apply to graphics, maybe not

ww

w.s

ite.

uo

ttaw

a.ca

/~el

sad

dik


9. Details: Usability Engineering

“A process whereby the usability of a product is specified quantitatively, and in advance. Then as the product is built it can be demonstrated that it does or does not reach the required levels of usability”

Partly engineering:• Design-evaluate-redesign

Partly science:• Experimentation methodology• Although not with full controls

ww

w.s

ite.

uo

ttaw

a.ca

/~el

sad

dik


Usability Engineering Steps

1. Pick ‘benchmark’ tasks• Simple tasks that can be repeated and performance measured

2. Pick usability metrics3. Set planned levels of usability4. Design initial interface using usability guidelines5. Analyze impact of design(s) using experiments

1. i.e. a new batch of users is measured running the benchmark tasks

6. If goals are achieved, stop7. Incorporate user derived feedback in design8. Go back to step 5

Major problem with usability engineering:• Benchmark tasks are rarely performed in a truly natural

environment

ww

w.s

ite.

uo

ttaw

a.ca

/~el

sad

dik


Some suitable usability metrics

• Time to complete task• Percentage of task completed per unit time• Ratio of successes to failures• Percentage of time spent dealing with errors• Percentage of competing products that have better speed

measures than our product• Number of repetitions of failed commands• Percentage of available commands used• Number of times user had to undo an action• Number of unnoticed errors• Number of times user did not use the expected method to

accomplish the task• Think time required for task

• i.e. ignoring system response time

• a good UI should lead user through system with minimal ‘cognitive load’

ww

w.s

ite.

uo

ttaw

a.ca

/~el

sad

dik


10. Details: Cognitive Walkthroughs

A form of predictive evaluation

Detailed reviews based on psychological theory, focusing on:• Goals a new user must form to execute a task• How well the system leads the user to form those goals

• i.e. how well the system supports the user

• The method is highly structured• Forms are provided to guide the evaluator

• More time consuming that ordinary heuristic evaluation• Less time consuming than experiments

ww

w.s

ite.

uo

ttaw

a.ca

/~el

sad

dik


Cognitive Walkthrough Steps

1. Choose a task to evaluate

2. Describe the task exactlya) First describe the task in one sentence

• Use simple language• The wording should be from a first-time user’s point of view

e.g. Record a newly-received item in inventory.b) Describe the initial state of the system

e.g. Main menu is displayedc) List the atomic actions needed to correctly perform the task, e.g.

1. Click on ‘add to inventory’ in the menu.

2. If you don’t know the part number, hit ‘return’ to perform look up the part number, then go to action 4.

3. Type the part number into the ‘part number’ field

4. Press tab

5. Type the number of items in the ‘Number’ field

6. Hit <return> or click on ‘Add’.

7. If the system prints out a bar-code sticker, affix it to the new item.d) Describe classes of users who may perform the task

e.g. Receiver - knows about inventory, but not yet about the system

ww

w.s

ite.

uo

ttaw

a.ca

/~el

sad

dik



e) Describe the ‘Goal Structure’ (or task structure) users would likely have in their minds before starting the task• High-level and system independent

• Indent subgoals/subtasks

• Note if there are actions for which the user has no goals, the system must stimulate the user to think of these goals by the time they must perform the task

• If different classes of user may have different goal structures, list these too.

e.g.

Record a received item in inventory

Started the inventory program

Enter the item

ww

w.s

ite.

uo

ttaw

a.ca

/~el

sad

dik



3. For each action specified in step 2c, do the following (I to IV):

I. Write down the goal structure

... that the user would need to have in order to perform the action correctlye.g. For action 4

• Record a received item in inventory

• Record the number of items

• Press tab

• Enter the number

• Cause the system to process the transaction

ww

w.s

ite.

uo

ttaw

a.ca

/~el

sad

dik



II. Verify that the user will have the correct goal structure• given their initial goals• given the system’s response to the previous action• Estimate the percentage of users who might have each of

the following possible problems:• Failure to add goals

• e.g. For action 2The system must make it clearly visible that pressing return with nothing entered will invoke a lookup mechanism

• Failure to drop goals• e.g. The user may have a goal to, notify the person who ordered the

parts This would not b needed if the system performs this automatically

• Addition of spurious goals• e.g. There may be a field marked ‘Description’

However this only needs to be filled in if the type of item is not in the database

ww

w.s

ite.

uo

ttaw

a.ca

/~el

sad

dik



• No-progress impasse• e.g. After adding an item, the system might just clear the screen ready

for another entry.• The user may think the transaction failed (i.e. goal not achieved)

• Premature loss of goals• e.g. The user enters an item and hits “return”• A message ‘transaction accepted’ is printed (meaning the transaction

has been started)• The user powers off the computer thinking the goal is reached• The system never got around to printing the label

ww

w.s

ite.

uo

ttaw

a.ca

/~el

sad

dik



III. Verify that the actions match the goalsPossible problems:

• Correct action doesn’t match goal• e.g. User wants to delete an item that was stolen.• Correct action is to select ‘add to inventory’ and specify a negative

number• System does not help user match the goal to the action

• Incorrect actions match goals• e.g. User wants to add a new type of item to inventory (for which no

items have yet been received)• Upon seeing ‘add to inventory’, user selects this incorrect menu item

IV. Verify that the user can physically perform the action Possible problems:

• Physical difficulties• e.g. recognizing an icon, holding down shift-ctrl-alt-a to perform a

command

• Time-outs• i.e. running out of time – the system gives up

ww

w.s

ite.

uo

ttaw

a.ca

/~el

sad

dik



• Objective of evaluation: Minimize malfunctions• Key questions:

• What is real task? Problems? Which is better?

• Met targets? Is it standard?• Evaluate throughout lifecycle!• Formative vs. summative evaluation• Pilot studies important• Use all techniques in a balanced approach• Use cost-benefit analysis to see if an expensive technique will

pay off

• Passive methods• Problem reporting

• Software logging

• Questionnaires/surveys

ww

w.s

ite.

uo

ttaw

a.ca

/~el

sad

dik



• Active methods• Traditional experiments

• Investigate a single UI element• Pick subjects• Independent and dependent variables• Hypotheses• Experimental designs:

• independent subject• matched subject (control for differences among subjects)• repeated measures (reuse subjects)

• Usability Engineering• Test realistic ‘benchmark’ task• Set targets for usability metrics• Evaluate-redesign-evaluate until targets met• Partly engineering, partly science

• Observation sessions (Videotaped Evaluation)• Study active use on realistic tasks• Think-aloud protocol on video• Co-Operative Evaluation involves dialogue

ww

w.s

ite.

uo

ttaw

a.ca

/~el

sad

dik



• Predictive evaluation: involve experts• Heuristic Evaluation: based on guidelines• Cognitive Walkthroughs: goals and actions

• Describe task, actions, users, goal structure• For each action, verify that users:

• ... add and drop goals as needed• ... don’t add unneeded goals• ... can tell when a goal is reached• ... don’t drop needed goals• ... can see what action to take• ... are not mislead into taking wrong action• ... have no physical difficulties with action

• Malfunction analysis• How manifested?

• Detected by: system, user• Undetected, inefficiencies

• What stage in interaction? When user...• Decides on goal? Specifies action?• Executes action? Interprets result?

• Level? (physical to task)• Why occurring?

• Users lacks: e.g. motivation, input, recall

Documents

Www.site.uottawa.ca/~elsaddik 1 Unit B-Introduction (c) elsaddik SEG 3210 User Interface Design & Implementation Prof. Dr.-Ing. Abdulmotaleb El Saddik