2005 IEEE International Professional Communication Conference Proceedings

0-7803-9028-8/05/$20.00 © 2005 IEEE.

Defining Usability: Quality of Use or Quality of Experience?

Niamh McNamara University College Corkn.mcnamara@ucc.ie

Jurek Kirakowski University College Corkjzk@ucc.ie

Abstract

This paper is concerned with the impact of changing conceptualizations of usability from quality of use to quality of experience. At present, there are many calls to expand the usability construct, although there is no clear consensus as to how this can be achieved. The lack of such a consensus might result in the usability construct being broadened to such a degree as to become meaningless. The implications of this for the evaluation of electronic consumer products are discussed.

Keywords: HCI, usability, electronic consumer products, user experience.

Introduction

The area of interest of this paper is evaluating the usability of everyday electronic consumer products such as mobile phones, DVD players, VCRs, washing machines, etc. An increasing number of products that people use in their daily lives contain embedded technology. This computerization of everyday life has created new challenges for the discipline of Human Computer Interaction (HCI). Although these devices were first introduced some 30 years ago, they remain difficult to use and grow increasingly complex. Therefore, the need for some form of usability evaluation is obvious.

Broadly speaking, the term usability refers to how easy a product is to use. It was first introduced in the 1980s to replace the cliché term user friendly[1]. While it is central to HCI, the discipline lacks a standard definition of the construct and consequently there are a number of competing perspectives on how it should be conceptualized. The dominant perspective defines usability as quality of use and this has informed much of the

research carried out on desktop systems. However, there is some debate concerning the applicability of this view of usability to new technology, including electronic consumer products. It is suggested that viewing usability as quality of use neglects the subjective aspects of using technology, such as emotional evaluations and considerations of a device’s aesthetic value, and thus the move should be made to conceptualizing usability as quality of experience. This paper discusses the implications of broadening the usability construct from quality of use to quality of experience and then offers a framework for evaluation.

Usability as Quality of Use The quality of use perspective on usability examines the interaction between the user and the product. It emerged as an alternative to a perspective that dominated early research and still exerts some influence today, which views usability as a product property. The underlying assumption in these early conceptualizations is that usability is dependent on the presence or absence of particular product features. However, such a conceptualization is static and situation-specific and does not guarantee that a product will be usable for all possible user groups and usage scenarios.

The quality of use perspective, however, hypothesizes that usability varies according to whois using the product, where it is being used, and the purpose for which it is being used. Collectively, the various aspects of the interaction that need to be considered when specifying usability are known as the context of use [2]. This view on usability is enshrined in the ISO 9241-11 definition of usability as “the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use” [6].

200

2005 IEEE International Professional Communication Conference Proceedings

This perspective views usability as an abstract but measurable construct that is central to the processes of Usability Engineering and User-Centered Design. The need for systematic measurement of usability requires operationalisation of the construct and a number of dimensions have been proposed. These include Effectiveness, Learnability, Flexibility, and Attitude [14]; Learnability, Memorability, Errors, and Satisfaction [11]; and Effectiveness, Efficiency, and Satisfaction [6]. An interesting feature of these operationalisations is that all include a subjective element. It is typically recommended that both objective and subjective data be collected when evaluating a product. Objective measures typically examine the accuracy and speed with which the product can be used while subjective measures assess user reactions and attitudes towards the product. Unfortunately, increasingly shorter product development timescales and the need for cost-effectiveness often means that subjective data is not gathered as its collection is regarded to be time-consuming and resource-intensive.

Those who characterize usability as quality of usenot only propose conceptual definitions of the construct by identifying specific dimensions to be included when measuring usability, they also endeavor to develop a theory of the construct by situating it in a wider model of user acceptance of technology alongside other concepts such as utility, functionality, likeability, and usefulness [14], [11]. Nonetheless, critics of the quality of use perspective argue that it ignores subjective aspects of technology use, such as ‘fun’ and ‘enjoyment.’ This is seen as damaging to the discipline as a whole, and has led to calls for a re-examination of the construct in general [7], [12], [15].

Usability as Quality of Experience Recently, quality of experience has entered the discussion.This perspective proposes to address aspects of technology usage that are omitted by the quality of use perspective. It should be noted that this is not the first time this perspective has been criticized for omitting important aspects of technology usage. Since Davis [3] proposed his Technology Acceptance Model (TAM), a number of authors have criticized HCI for focusing exclusively on usability and neglecting other factors that might influence technology acceptance and usage. According to Davis [3], considering a product’s perceived usefulness may be more important in

determining whether or not that product will be used than examining its perceived ease of use orusability. However, there is one important difference that distinguishes the concepts proposed by proponents of TAM from those proposed by advocates of the quality of experience approach. In Keinonen’s terms, concepts such as usability, utility, and usefulness are factors that influence extrinsic motivation to use technology. Those who call for a focus on the user experience propose to investigate how individuals are intrinsicallymotivated to use technology [8] and so concepts that reflect the more subjective aspects of technology usage such as engagement, pleasure, presence, and fun are receiving more attention.

There are three main approaches to investigating quality of experience evident in the literature. The first approach involves adding new dimensions to the usability construct. In a sense, this involves revising the operationalisations of usability discussed earlier to include more subjective dimensions. For example, Logan [9] suggests that usability should be split into behavioral usability,which refers to “the ability to complete some functional or goal-directed task within a reasonable time” and emotional usability, which refers to “the degree to which a product is desirable or serves a need beyond the traditional functional objective.” Therefore, this approach seeks to broaden the usability construct to include additional concepts rather than discarding it completely and starting afresh.

The second approach involves the proposal of new concepts that are hypothesized to be important to consider when designing technology – but that are distinct from usability. For example, Jordan [7] contends that usability is “inherently limited” by placing too much emphasis on cognitive issues. Rather than exploring how the use of technology elicits positive feelings in the user, HCI has concentrated on the avoidance of negative emotions [7]. He maintains that the pleasure-based approach to design that he proposes, “(encourages) a holistic view of the user, judging the quality of a design on the basis of the wider relationship between a product and the people for whom it is designed.” Jordan describes a three-level hierarchy of consumer needs, based on Maslow’s [10] Hierarchy of Needs, with Functionality at the first level, Usability at the second, and Pleasure at the third. Pleasure is here defined as “the emotional, hedonic and practical benefits associated with

201

2005 IEEE International Professional Communication Conference Proceedings

products.” Jordan maintains that the consumer’s most basic need is for products to work. After this, people expect products to be easy to use. Finally, once people become accustomed to usable products, they want something more, i.e. pleasurable products. Such products can be created by linking product properties to particular user emotions [7].

Finally, a third approach, proposed by Wright & McCarthy [16], takes a more theoretical and philosophical look at the concept of experience. They argue that experience is an under-developed concept and is often used without a deeper understanding of what the term actually means [16]. While other approaches (such as that outlined by Jordan [7]) attempt to design a particular user experience by linking product properties to user emotions, this approach advocates the use of structured techniques to talk about experience. To this end, Wright & McCarthy [16] propose a framework with which to analyze experience based on the work of philosophers John Dewey and Mikhail Bakhtin. This framework gives researchers a method of talking about experience, and consists of two parts: describing an experience and making sense in experience. It criticizes attempts to design an experience and maintains that experience cannot be reduced to its fundamental elements [16].

Clearly, there are many strands to the “New Usability” involving a number of new approaches and concepts. There is considerable debate, however, as to whether these concepts should complement usability evaluations or, indeed, replace such evaluations.

Going beyond usability – the concept of experience

Although there are many calls to go beyond usability, no one quite knows what beyond is [9]. This situation is made all the more difficult by the lack of any standard definition of usability. Thus, the introduction of the concept of experience adds further complexity to an already complex construct.

This is particularly evident in attempts to broaden the usability construct by including concepts such as aesthetics, image, impression, and ambience, among others. There is little, if any, explanation of how these concepts are related to each other or to usability with the result that the usability construct

could be broadened to such a degree as to become meaningless or else narrowed severely to the same ultimate effect.

It is suggested here that usability is a concept distinct from that of experience. Indeed, this is similar to the position that Jordan [7] takes in his proposed hierarchy of consumer needs. However, his operationalisation of the pleasure construct is rather narrow and shares many similarities with early conceptualizations of usability as a productproperty in the way that it links product properties to user emotions. Conceptualizing usability in this way is considered inappropriate and it is unlikely that a conceptualization of experience as dependent on particular product attributes will prove any more successful. As Wright & McCarthy [16] argue, experience is as much about what the individual brings to the interaction as it is about the product itself. The concept of experience is being increasingly used in the HCI literature, however without a deeper understanding of what exactly constitutes experience it will remain little more than a buzzword in the same vein as “user-friendly” in the 1980s [5].

Implications for Evaluation Clearly, changing conceptualizations of the usability construct have implications for the evaluation of new technology, including electronic consumer products. Investigating the wider relationship between people and technology would no doubt be beneficial to the design of technology and the discipline of HCI as a whole, as it is clear that there are aspects of usage that need to be considered because the way in which technology is used changes. Nonetheless, the fact remains that such products are also difficult to use and grow increasingly complex, suggesting that calls for the abandonment of the usability construct are somewhat premature.

It is suggested here, in contrast to Jordan’s [7] hierarchical view, that there are 3 aspects that need to be considered simultaneously when designing and evaluating technology. This is illustrated in Fig. 1. Each of these 3 areas asks a different question about technology usage using a different language of discourse.

Functionality is a technical issue. In this instance, the goal of evaluation is to answer the question “what will the product do?”

202

2005 IEEE International Professional Communication Conference Proceedings

Figure 1. 3 areas of technology usage

Usability is a user issue and therefore the product needs to be tested with real users. The criteria of support and goal facilitation are important here in answering the question, “can I make the product do what I want it to do?” Finally, examining the userexperience investigates the wider relationship between the product and the user in order to consider the individual’s personal experience of using it. The question asked in this area may be “how do I relate to this product?”

These areas are not arranged in the form of a hierarchy. As such, a presentation not only implies that one aspect of usage is more important to consider than others, but also it implies that one level builds on the other, which it does not. For instance, one could examine the user experience of using a product regardless of whether that product is usable or not. While the various aspects might influence each other to some extent, for example, the implementation of device functions impacts on the usability of the product and the usability of the product impacts on the experience of using it, independent evaluations can, and should, be carried out in each area. According to this functionality or experience should not be incorporated into usability evaluations, nor should they be included in definitions of the construct.

It is important to note that each area of investigation incorporates its own unique set of methodologies and the significance of choosing an appropriate methodology when carrying out an evaluation in any of the 3 areas cannot be overemphasized. The disadvantage with some

approaches belonging to the quality of experienceperspective outlined earlier is the use of inappropriate methods to investigate the user experience. It should be clear that one could not use established usability evaluation methods (UEMs) to assess the user experience. (Likewise, one cannot use UEMs to assess the product’s technical quality). It is vital that experience is not reduced methodologically to usability in just the same way that it would be a “category mistake” [13] to attempt to measure product functionality in UEM terms.

Conclusion

It is argued here that all areas of technology usage (i.e. functionality, usability and user experience) should be considered in the design and evaluation of electronic consumer products. The move suggested by some from conceptualizing usability as quality of use to quality of experience has some shortcomings, not least because the concept of user experience is still conceptually vague. It may be that separating the 3 areas will enable greater clarity of definition. Therefore, it is argued here that usability should be conceptualized as the quality of use of a product which reflects an individual’s ability to successfully use a product to achieve a particular goal. Furthermore, it is recognized that advances in technology change the relationship between people and technology; therefore, there are other aspects of using technology that need to be considered during design and development, as well as during evaluation. This necessitates the inclusion of structured approaches and frameworks for investigating experience into the discipline’s repertoire, as well as further research into the concept of experience. Yet it is argued here that such investigations should be undertaken in conjunction with technical and usability evaluations rather than replacing or broadening them.

References

[1] N. Bevan, J. Kirakowski & J. Maissel, “What is usability,” in Proc. 4th Int. Conf. On HCI, Stuttgart: 1991.

[2] N. Bevan & M. MacLeod, “Usability measurement in context,” Behavior and Information Technology, vol. 13, pp. 132-145, 1994.

203

2005 IEEE International Professional Communication Conference Proceedings

[3] F. D. Davis, “Perceived usefulness, perceived ease of use and user acceptance of information technology,” MIS Quarterly, vol. 13, no. 3, pp. 319-340, 1989.

[4] S. H. Han, M. H. Yun, J Kwahk, & S. W. Hong, “Usability of consumer electronic products,” International Journal of Industrial Ergonomics, vol. 28, pp. 143-151, 2001.

[5] M. Hassenzahl, M. Burmester & F. Koller, (2004). Usability Engineering: Quality doesn’t happen by accident. Available: http://www.tu-darmstadt.de/fb/fb3/psy/soz/veroeffentlichungen.mh/Mti.hassenzahl.pdf

[6] ISO 9241, Ergonomic Requirements for Office Work with Visual Display Terminals: Part 11: Guidance on Usability, 1998.

[7] P. W. Jordan, Designing Pleasurable Products: An Introduction to the New Human Factors,London: Taylor & Francis, 2000.

[8] T. Keinonen, One-Dimensional Usability: Influence of Usability on Consumers’ Product Preference. Netherlands: University of Art & Design. 1998.

[9] R. J. Logan, “Behavioral and emotional usability: Thomson consumer electronics,” in Usability in Practice. M. E. Wiklund, Ed. New York: AP Professional, 1994, pp. 59-82.

[10] A. H. Maslow, “A theory of human motivation,” Psychological Review, vol. 50, pp. 370-396, 1943.

[11] J. Nielsen, Usability Engineering. Boston: Academic Press. 1993.

[12] D. Rijken & B. Mulder, “Information ecologies, experience and ergonomics,” in Usability Evaluation in Industry. P. W. Jordan, B. Thomas, B. A. Weerdmeester & I. L. McClelland, Eds. London: Taylor & Francis, 1996, pp. 49-58.

[13] G. Ryle, The Concept of Mind. London: Flutchinson. 1949.

[14] B. Shackel, “Usability – Context, framework, definition, design and evaluation,” in Human Factors for Informatics Usability, B. Shackel & S.

Richardson, Eds. Cambridge: Cambridge University Press, 1991, pp. 21-37.

[15] P. Thomas & R. D. Macredie, “Introduction to the new usability,” ACM Trans Computer-Human Interaction, vol. 19, no. 2, pp. 69-73, 2002.

[16] P. C. Wright & J. C. McCarthy, “Making sense of experience,” in Funology: From Usability to User Enjoyment. M. Blythe, A. Monk, C. Overbeeke & P. C. Wright, Eds. Dordrecht: Kluwer, 2003, pp. 43-53.

About the Authors

Niamh McNamara graduated with a 1st Class Honors degree in Applied Psychology from University College Cork. Her dissertation was an investigation of the usability of the WAP (Wireless Application Protocol) service. Niamh has undertaken research into the quality of services provided by UCC and evaluating the usability of a website for adults with limited literacy skills. She is currently pursuing a PhD on the development of a tool to evaluate user satisfaction with electronic consumer products.

Jurek Kirakowski’s speciality is quantitative measurement in Human-Computer Interaction and he has contributed numerous books, articles, and workshops to this theme. His major research goal has been to show, and indeed prove, how the quality of use of Information Technology products can, and should be, quantitatively measured in an objective manner. Dr. Kirakowski and his group have contributed the SUMI (Software Usability Measurement Inventory), and the WAMMI (Website Analysis and Measurement Inventory) questionnaires which are by now ‘de-facto’ standards in their respective areas. His personal home page is found at http://www.ucc.ie/hfrg/jk

204