Qaiser S. Durrani FAST-NU, Lahore Workshop on Usability Engineering Feb 21-23, 2011 at SEECS NUST

Agenda Usability Engineering? Why we need it? What are its measures? Where UE fits in the SDLC Can we integrate or map UELC with SDLC? Experience and Emotional Measures Role? Case Study Current practices in Software Industry with respect to UE

W hy Usability Engineering? Functional perspective User perspective

Usability the capability to be used by humans easily and effectively quality in use the effectiveness, efficiency, and satisfaction with which specified users can achieve goals in particular environments Context dependent (shaped by the interaction between tools, problems, peoples) A process through which usability characteristics are specified and measured throughout the software development lifecycle.

Key Research Questions in HCI How to work with and improve the usability of interactive systems? Guidelines for improving the usability of systems? Methods for predicting usability problems? Techniques to test the usability of systems? Discussions on how to measure usability

Neglecting Usability Engineering

Usability into Software Development When integrating usability into the system design process, early focus on users and tasks, empirical measurement, and iterative design principles are suggested This integration, however, is not a trivial task, as numerous obstacles have been reported First of all, introducing a new method into a software development organization is typically a delicate problem User-centered design techniques have been reported to remain the speciality of visionaries, isolated usability departments, enlightened software practitioners, and large organizations, rather than the everyday practice of software developers

Usability Engineering and Experience Design

Models for Usability Engineering Lifecycle Star Lifecycle Model ISO 13407 Model Usability engineering lifecycle by Deborah J. Mayhew

Usability Engineering Lifecycle

Requirements Analysis Phase User Profiling Cognitive & Non-Cognitive measures Task Analysis SW/HW/Environment Constraints General Design Principals Usability Goals

Design, Development & Evaluation Conceptual Level Design Detail Level Design Screen Standards Iterative Evaluation

Usability Activities

Adaptation of Usability Activities into Software Engineering Development Process

Allocation of Usability Techniques to Development Activities

Shneidermans Golden Rules R1:Strive for consistency R2:Offer shortcut R3:Give effective feedback R4:Reduce Short term memory load R5:Provide reversal of actions R6:Design Dialogues to yield closure R7:Provide locus of control

Practices - MEASURING USABILITY (Case study of 180 projects) Measures of effectiveness Measures of Efficiency Measures of Satisfaction

Measures of Effectiveness Binary task completion Accuracy Recall Completeness Quality of outcome Experts assessment

Comments 1- 22% of the studies reviewed do not report any measure of effectiveness nor do these studies control effectiveness. Frkjr et al. argued that the HCI community might not succeed in trying to make better computing systems without employing measures of effectiveness in all studies 2- Research shows that measures of the quality of the outcome of the interaction are used in only 16% of the studies. For example, experts assessment of work products seems a solid method for judging the outcome of interaction with computers and has been used in a variety of fields as an indicator of the quality of work products, for example with respect to creativity. Yet, in this sample only 4% of the studies use such measures

Comments 3- New kinds of devices and use contexts require new measures of usability. Especially, it has been argued that the notion of task underlying any effectiveness measure will not work in emerging focuses for HCI, such as home technology 4-A number of studies combine usability measures into a single measure, report the combined values, and make statistical tests on the combinations

Measures of efficiency Time Input rate Mental effort Usage patterns Communication effort Learning

Comments 1-Some of the efficiency measures are obviously related to the quality of interactive computer systems, because they quantify resources (e.g., time or mental effort) that are relevant in many contexts for many users 2- A second comment on the studies reviewed pertains to the measurement of time. A surprising pattern apparent from Table is that while objective task completion time is measured by 57% of the studies, little attention is paid to users experience of time However, in this sample of 180 studies, only one study measures directly subjective experience of time

Comments 3- The reviewed studies differ in how task completion times, and efficiency measures in general, are reasoned about. In the ISO definition of usability and in most of the studies reviewed, time is considered a resource of which successful interfaces minimize consumption However, in a handful of studies higher task completion times are considered as indicators of motivation, reflection, and engagement

Comments 4- A striking pattern among the studies reviewed is that few studies (5) concern learning of the interface. Only five studies measure changes in efficiency over time 5- In the studies reviewed, the median time of working with the user interfaces evaluated was 30 min

Measures of Satisfaction Standard questionnaires Preferences Satisfaction with the interface User attitudes and perceptions

Comments 1- The measurement of satisfaction seems in a state of disarray. A host of adjectives and adverbs are used, few studies build upon previous work, and many studies report no or insufficient work on the validity and reliability of the instruments used for obtaining satisfaction measures Another indication of the disarray is in the limited use of standardized questionnaires

Comments 2- A second comment on the satisfaction measures used is that studies vary greatly in the phenomena that are chosen for objective performance measures and those that are investigated by asking subjects about their perceptions and attitudes. One question arises when users perception of phenomena is measured when those phenomena perhaps more fittingly could have been assessed by objective measures 3- The review shows that in practice subjective satisfaction is taken to mean a questionnaire completed after users used the interface. Only eight studies (4%) measure satisfaction during use without using questionnaires

CHALLENGES IN MEASURING USABILITY

Subjective and objective measures of usability Measures of usability concern users perception of or attitudes towards the interface, called subjective usability measures Other measures concern aspects of the interaction not dependent on users perception called objective usability measures Such a distinction has been argued to simplify the nature of measurement in science Suggest using the distinction to reason about how to choose usability measures and find more complete ways of assessing usability Measures may lead to different conclusions regarding the usability of an interface

Measures of learnability and retention Particularly measures of efficiency, we find it relevant to compare them to recommendations on how to measure usability The well-known textbook by Ben Shneiderman (1998, p.15) recommends measuring (1) time to learn, (2) speed of performance, (3) rate of errors by users, (4) retention over time, and (5) subjective satisfaction. Nielsen (1993, p. 26) similarly recommends measuring (a) learnability, (b) efficiency, (c) memorability, (d) errors, and (e) satisfaction Most of the reviewed studies follow part of the recommendations by measuring task completion time (points 2 and b above), accuracy (points 3 and d), and satisfaction with the interface (points 5 and e): 92% of the studies measure at least one of these; 13% of the studies measure all three

Measures of learnability and retention The majority of studies make no attempt to measure learnability or retention This challenge is most relevant for studies or research addressing systems that users should be able to learn quickly or that will be intensively used Overall, usability studies could put more emphasis on measures of learning, for example by measuring the time needed to reach a certain level of proficiency In addition, measures of the retention of objects and actions available in the interface (i.e., the ability of users to come back and successfully use the interface) are important in gaining a more complete picture of usability

Measures of usability over time The studies reviewed show that users typically interact only briefly with interfaces under investigation; as mentioned earlier the median duration of users interaction was 30 min; only 13 studies examined interaction that lasts longer than five hours The brief period of interaction in the studies reviewed explains the lack of focus on measures of learning and retention The observation also suggests that we know little about how usability develops as the user spend more time interacting with the interface and how tradeoffs and relations between usability aspects change over time From research, we need a more full understanding of how the relation between usability aspects develops over time

Extending, validating and standardizing measures of satisfaction The disarray of measures of satisfaction presents special challenges One is to extend the existing practice of measuring satisfaction almost exclusively by post-use questions; another is to validate and standardize the questions used Validation may be achieved through studies of correlation between measures

Micro and macro measures of usability Usability at a micro level Such measures cover tasks that are usually of short duration (seconds to minutes), has a manageable complexity (most people will get them right), often focus on perceptual or motor aspects (visual scanning, mouse input), and time is usually a critical resource Usability at a macro level Such measures cover tasks that are longer (hours, days, months), are cognitively or socially complex (require problem-solving, learning, critical thinking, or collaboration)

A working model for usability measures and research challenges

Affective Requirement The need to make something fun, engaging, or enjoyable is usually not considered in requirements elicitation Software requirements for these and other affective factors are never truly captured in an official manner Juran is credited with coining the phrase "fitness for purpose If a system is intended to be a leisure product then the fitness for purpose must also extend to affect

Rebirth of Affect in Design The idea of affect is not old but affect has re-emerged as a potentially desirable design characteristic One of the visionaries of this re-emergence was Robert Glass from Sun Microsystems, who said: If youre still talking about ease of use then youre behind. It is all about the joy of use. Ease of use has become a given its assumed that your product will work. (Glass, 1997)

Summary of research into affective factors

Exploring AffectTheories Three theories have each been said to contribute to computer game enjoyment Usability: In ISO 9241-11 (ISO, 1998), usability is characterized as consisting of three elements: effectiveness, efficiency, and satisfaction Grice (2000) attempted to apply these three elements to computer game design His hypothesis was that computer games that were enjoyable will have high levels of efficiency, effectiveness, and satisfaction Some minor experiments conducted under his supervision seemed to indicate that this hypothesis was true

Exploring AffectTheories Flow: Csikszentmihaly describes flow as the holistic sensation that people feel when they act with total involvement In the state of flow, actions flow without conscious intervention by the actor The term flow was used because people in this state often said that they were in the flow of [the activity]. the characteristics of flow-inducing activities are: must feel capable of completing the task must have the ability to concentrate on task clearly recognizes the goals of the task receives immediate feedback about task performance has a sense of control over their actions has the sense of time altered: hours can seem like minutes

Exploring AffectTheories Heuristics for internally motivating interfaces: Malone (1983), in agreement with Csikszentmihaly, believes that fun and enjoyment only arise from activities that are intrinsically motivated Computer games are thought to be played because of intrinsic motivation, with no expectation of a reward other than the activity itself Malone and Lepper (1987) developed seven heuristics for the design of intrinsically motivated interfaces

Exploring AffectTheories The 4 major heuristics are: Challenge- multi-layers of challenge so that the user will feel initial success and continue to see improvements Curiosity- believe that their knowledge structures (or skills) are incomplete or inconsistent Control- interface should make the user feel that the outcomes are determined by the users own actions Fantasy- evoke mental images of physical or social situations Other minor are Competition, Cooperation, Recognition

Results The results being referred to are the learnability and losing time reasons Loss of Time Learnability

Measures of specific attitudes towards the interface (Experience Design) from 180 projects

Current Usability Practices in Pakistan Software Industry

Basic Software Industry Data Number of SW industry surveyed: 26 Number of respondents: 35 Project Type: Multiple type from Web to IS

Research Questions Does organization include estimates for usability activities in planning phase? Does organization involve users during SDLC phases? If yes then what kind of user involvement it has? (a) Are usability activities integrated into requirement phase of SDLC? (b). Are usability activities integrated into design phase of SDLC? (c) Are usability activities integrated into implementation phase of SDLC? (d) Is usability testing done in an organization? Does an organization collect feedback from users for a product? Does an organization calculate return on investment for the usability activities? Are organizations intended to introduce or enhance the UELC activities in SDLC?

Fig1:Usability Activities in Planning PhaseFig2:User Involvement in SDLC Fig3:User Involvement in SDLC Phases

Fig4: Usability Requirements Fig5: User Profile Fig6: User Contextual Inquiry

Fig9: Usability Roles Fig7: Usability Goals Fig8: User Interface Development platform

Fig10: Screen Design Standards Fig11: Design Flexibility Fig12: Usability Roles

Fig13: Detailed Design of User interface Fig14: Usability Testing

Fig15: User Feedback Fig16: User Experiences

Fig17: User Feedback Fig18: ROI Calculation

Challenges in Measuring Usability Subjective and objective measures of usability Measures of learnability and retention Measures of usability over time Extending, validating and standardizing measures of satisfaction

Recommendations Developers must consider user interaction from the beginning of the development process. Practice of Usability Testing Practice of Cost-Justifying Usability tasks Dont try to do a full-scale usability process from the beginning.

References Kasper Hornbk, Int. J. Human-Computer Studies (2006),Current practice in measuring usability: Challenges to usability studies and research Xavier Ferre, Integration of Usability Techniques into the Software Development Process Juho Heiskari, Marjo Kauppinen, Mikael Runonen, Tomi Mannisto, Bridging the Gap Between Usability and Requirements Engineering, 2009 17th IEEE International Requirements Engineering Conference Todd Bentley, Lorraine Johnston, Karola von Baggo, AWRE2002, Putting Some Emotion into Requirements Engineering Samia Asloob, Qaiser S. Durrani, Usability Engineering Practices in SDLC, Technical Report (2010), FAST-NU, Lahore

Questions?

Bottom Line benefits Increased Productivity Decreased user training Decreased user errors Decreased need of on-going technical support Incorporating business and marketing goals while catering to the user needs (especially for Mobile, Web and Gaming applications)

Time Constraints for the Application of Usability Activities and Techniques

Subjective and objective measures of usability Challenges in research are to develop subjective measures for aspects of quality-in-use that are currently measured by objective measures, and vice versa, and evaluate their relation In studies of usability, we suggest paying special attention to whether subjective or objective measures are appropriate, and whether a mix of those two better covers the various aspects of quality- in use

Definition of Process Increments defined seven deltas in order to get a better match with the general stages of an iterative software development process D1: Early Analysis D2: Usability Specifications D3: Early Usability Evaluation D4: Regular Analysis D5: Interaction Design D6: Regular Usability Evaluation D7: Usability Evaluation of Installed Systems

Affective Requirement Same functional requirements, underwent a similar design process by the same designers, yet the need to convey a different affective response greatly changed the entire product Given that requirements give the constraints on how a system should behave, then it is important to see that affective requirements are considered a valid category of requirement Accepting that affective factors make valid requirements raises the following questions: How does an organization elicit and document affective requirements? How does an organization design to meet affective requirements? How does an organization validate that the design elicits the required affective response?

Motivations Research focus on how to measure usability has three motivations: First, what we mean by the term usability is to a large extent determined by how we measure it Second, usability cannot be directly measured so, find aspects of usability that can be measured Which measures of usability to select is consequently central in many approaches to the design and development of user interfaces

Studies of correlations between measures A weak understanding of the relation between usability measures gives rise to many of the issues With a better understanding, we could make more informed choices about which usability measures to employ Studies of correlation between measures may improve this understanding by informing us whether our measures contribute something new and what their relation are to other aspects of usability There is need for a better understanding of the relation between usability measures, for which studies of correlations between measures would be one contribution

Documents

Qaiser S. Durrani FAST-NU, Lahore Workshop on Usability Engineering Feb 21-23, 2011 at SEECS NUST