Dynamics of user experience: How the perceived quality of mobile phones changes over time

Dynamics of user experience: How the perceived quality of mobile phones changes over time

Margeritta von Wilamowitz-Moellendorff

Social psychology and decision-making

Darmstadt University of Tech-nology

Alexanderstraße 10, 64283 Darmstadt, Germany

[email protected]

Tel: +49 (0) 06151 - 163066

Marc Hassenzahl Economic Psychology, Psychol-ogy of Work- and Social Behav-

ior Campus Landau

Im Fort 7 76829 Landau, Germany

[email protected]

Axel Platz Siemens AG, Corporate Tech-

nology, CT IC 7 User Interface Design, Otto-Hahn-Ring 6, 81730 Munich,

Germany [email protected]

ABSTRACT The way we use and experience an interactive product is a dynamic process. Surprisingly, only a very small number of studies in the context of Human-Computer Interaction (HCI) attempt to track and explain changes in behavior and experience over time. The present paper introduces CORPUS (Change Oriented analysis of the Relationship between Product and USer), an interview technique to re-construct changes in user experience over a period of more than one year. Preliminary results (addressing the domain of mobile phones) revealed characteristic changes. Whereas pragmatic perceptions remained stable (utility) or even im-proved over time (usability), hedonic perceptions (stimula-tion, beauty, communicate identity) exclusively deterio-rated. Stimulation showed an accelerated deterioration, caused by increasing habituation, whereas deterioration of beauty and identity was induced by comparisons with other peoples' products. All in all, we argue for a more dynamic perspective to user experience.

Author Keywords Mobile phone, quality dimensions, time, user experience, hedonic qualities, interview, change incidents

ACM Classification Keywords Categories and Subject Descriptors: H.5.m [Information interfaces and presentation]: Miscellaneous

General Terms: Design, Human Factors

INTRODUCTION Experiencing an interactive product is a dynamic encounter. Usage episodes stretch over time, each made of a series of single moments. At each moment, we may love the product or hate it; we may experience a usability problem or may find a useful new function; we may feel excited and stimu-lated or utterly bored. Moreover, as long as an episode con-sists of more than one moment, we may experience love, hate, excitement or boredom – all within a single usage episode.

This change over time – the dynamics of experience – is an important aspect, which is, however, virtually neglected by current approaches to user experience – or even more broadly – Human-Computer Interaction (HCI). The study of differences between novices and experts [e.g., 6] may be the closest to this objective. However, current approaches to user experience, which emphasize the importance of so-called hedonic needs or benefits (e.g., novelty, change, self-expression) as a complement to pragmatic needs or benefits (utility, usability), almost never address dynamic issues [4].

The present study's aim is to bring together the idea of tem-poral aspects and different quality dimensions, that is, pat-terns of change in pragmatic and hedonic quality percep-tions. It takes up questions such as how the perception of a product (for example as usable) changes over time and which processes or incidents are responsible for change.

The paper starts with a brief review of studies in HCI ad-dressing time as a variable. After this, the CORPUS (Change Oriented analysis of the Relationship between Product and USer) interview technique is introduced and preliminary results of a study with users of mobile phones are presented.

UX WS NordiCHI'06: COST294-MAUSE

74

CHANGE-ORIENTED STUDIES IN HCI Few studies in HCI explicitly addressed time as an impor-tant variable [e.g., 1, 5].

The most prevalent are cross sectional studies, which cover only one point in time, but with participants with different levels of experience. For example, Prümper and colleagues [6] studied the way novices and experts handle errors when using the computer for standard office tasks. They meas-ured the number of errors and error handling time. Interest-ingly, experts did not commit fewer errors compared to novices; however, they spent less time with handling them. This is just one example for a cross-sectional study, which compares different groups of users (novices and experts) to address change in user behavior. Albeit the most prevalent, this cross-sectional approach is problematic. Change is not directly studied, but only inferred from differences between groups of participants. As long as not the same participants are monitored over time, one cannot differentiate between differences due to time or due to interpersonal variation

Longitudinal studies are more suitable to study change. They cover a particular period of time, with two or more observations or measurements taken from them same par-ticipants. They can be further distinguished by the time pe-riod covered, the frequency of observation/measurement and the number of participants. The present paper focuses primarily on the time period covered. In the context of HCI, it is useful to distinguish between a micro perspective (e.g., an hour), a meso perspective (e.g., 5 weeks) and a macro perspective, with a scope on years of use and the idea to map the whole product lifecycle.

A typical study in HCI with a micro perspective is a usabil-ity test. Participants work on a series of tasks, often for an hour or two, while being observed constantly and maybe questioned repeatedly. Although interesting, it seems ques-tionable whether the time period covered by a micro per-spective is long enough to really witness changes in behav-ior or judgment.

Studies with a meso perspective are already becoming rare. An exception is a study by Mendoza and Novick [5], who observed 32 users of software for creating websites over a period of eight weeks. They focused on the general level of frustration and validated a categorization scheme for frus-trating usage episodes. The level of frustration decreased over time. In addition, user's reactions to frustrating epi-sodes changed markedly. Whereas in the beginning, partici-pants tended to ask other colleagues more often, later they frequently found ways to fix the underlying problem them-selves. These results already hint at dynamic processes.

Studies with a macro perspective are nearly non-existent in HCI. This is certainly due to the effort, which has to be put into these studies. Sometimes macro data is available in the field of remote usability, where usability problems are col-lected by so called "report incident buttons" over a longer period of time [1]. Critical incidents provide information about problems encountered while working on real tasks in

a working environment and are therefore an important source of information. However, such an approach exclu-sively focuses on usability problems, thereby excluding hedonic quality aspects important for user experience.

This overview of studies in HCI demonstrates the lack of studies, taking change over time and hedonic quality di-mensions into account. In addition, it highlights general problems with longitudinal study, such as the enormous investment for studies covering months or even years of usage. In the following section, we suggest a timeline ori-ented interview technique as a viable alternative to longitu-dinal studies.

THE 'CORPUS'-INTERVIEW AS AN ALTERNATIVE TO LONGITUDINAL STUDIES The CORPUS (Change Oriented analysis of the Relation-ship between Product and USer) interview technique fo-cuses on change in the nature of the user-product relation-ship. It addresses 'expert' users/owners (having used the product for at least 1 to 2 years) and combines the charac-teristics of two distinct interview forms: partly standardized and episodic interviews.

The more standardized part of the interview is directed at identifying change in the perception of different quality dimensions during the last one to two years. It starts from the now and goes back to the beginning of the relationship with the product. If change occurred, its direction and shape is assessed (e.g., accelerated improvement, steady deteriora-tion). In addition, the interview switches to a more open, episodic technique to further explore underlying conditions and causes of change. Specifically, participants are in-structed to tell little stories related to the change. These narratives, so-called "change incidents", give additional insights into underlying causes and conditions for change.

To guarantee broad coverage in line with current ap-proaches to user experience, CORPUS elicits change in five distinct quality perceptions (dimensions). They are primar-ily based on Hassenzahl's [2, 3] model of user experience. He distinguishes pragmatic and hedonic quality aspects. Pragmatic dimensions are directly related to task accom-plishment. They address "do goals", such as "making a telephone call", "sending an SMS". In contrast, hedonic dimensions are self-referential. They address "be-goals", such as "being cool", "being fashionable". Recently, beauty became of interest as a further quality dimension [7]. Al-though not directly addressed in Hassenzahl's model, beauty can be understood as a hedonic dimension as well [4].

Specifically the following two pragmatic and three hedonic dimensions were addressed:

- Utility (UT): the ability of a product to provide the nec-essary functions for given tasks.

- Usability (US): the ability of a product to provide the functions in an easy and efficient way.


75

- Stimulation (S): the ability of a product to surprise, to foster curiosity and to provide opportunities for the per-fection of knowledge and skills

- Beauty (B): the ability of a product to evoke a feeling of “beauty”

- Communicate identity (I): the ability of a product to communicate self-serving symbols to relevant others

Additionally, a global evaluation (G) of the product was included as a sixth dimension.

In the structured part of the interview, participants were shown a graph with a timeline as x-axis and a 10-point scale ranging from "not at all" to "very much" as the y-axis. They were asked about their perception of the product on a par-ticular quality dimension (e.g., usability) at the moment ("How much usability does the product has on a 0 to 10 scale?"). This was followed by the same question, however, referring to the beginning of usage. If the two assessments (current and initial perception) differed, the particular shape of change was further explored, for example, by asking, "whether the change occurred in a specific month", "more at the beginning" etc.

The interview started with the global evaluation (omitting the elicitation of change incidents), followed by the five quality dimensions in random order.

STUDY Although CORPUS accommodates to all interactive prod-ucts, we ran our first study in the domain of mobile phones. For mobile phones both pragmatic and hedonic attributes are considered important and the average product lifecycle is between 18 to 24 months. Later studies will use other products (product categories) and may, thus, be able to re-veal product or product category specific patterns of change over time.

Eight individuals participated in the study (7 female). The participants were students of psychology. They received no compensation for participation.

All participants owned and used their mobile phone for at least 12 months. Median usage was 20 months. An inter-view took about 45 minutes.

Concerning change over time, three important questions can be answered with the help of CORPUS data:

1. Trend: Is there a general trend for each quality di-mension? Which quality perceptions improve over time, which deteriorate?

2. Shape: What is the shape of the trend? Is it steady? Or does it predominantly take place in the begin-ning or at the end?

3. What are the causes for change? Is it possible to identify particular incidents, which lead to change?

Trend: stable, improving or deteriorating In general, the perception of a quality dimension can remain stable (e.g., the product appears today as useful as in the beginning), it can improve (e.g., it appears more useful to-day than in the beginning), or deteriorate (e.g., it appears less useful today than in the beginning)

Table 1 (printed in italics) shows the absolute frequency of the different trends for each quality dimension and the global rating. The gray shade signifies cells with the highest absolute frequency for each dimension.

Trend / Shape Pragmatic Hedonic G UT US S B I G stable 8 4 2 3 1

improving 4 1 1 accelerated 4 1 steady 1 jump

deteriorating 8 5 5 6 accelerated 7 1 steady 1 1 2 2 jump 4 2 4

Table 1. Absolute frequency of trends and shape for change in quality perceptions

Overall, pragmatic dimensions remained stable or im-proved, whereas hedonic dimensions deteriorated. Specifi-cally, utility (UT) remained stable; usability (US) remained stable or improved. Stimulation (S) uniformly deteriorated. Beauty (B) and identity (I) deteriorated (5) or remained stable. Only a single individual reported an improving beauty-

Interestingly, the global rating deteriorated as well (6 out of 8). From this, we may tentatively conclude that in the case of mobile phones global evaluation was rather driven by hedonic than pragmatic considerations

Shape: steady, accelerated, or jumping The CORPUS technique asks participants to assess the shape of change for each quality dimension (if change oc-curred). Three general shapes emerged: accelerated (i.e., sharp improvement/deterioration at the beginning of usage, 0-2 months), steady (i.e., continuous improve-ment/deterioration), or jump (i.e., sudden increase or de-crease due a specific event)

Table 1 shows the absolute frequency of the three different shapes for improvement and deterioration.

In the four cases, where a pragmatic dimension (namely usability) changed over time, participants reported an accel-erated improvement. The hedonic dimensions considerably differed. Stimulation revealed an accelerated deterioration,


76

whereas the deterioration of beauty was predominantly jumping, i.e., related to specific events. For identity all pos-sible shapes were observed.

Figure 1 summarizes the typical trend and shapes of the five quality dimensions on the basis of the mean elicited ratings.

0

2

4

6

8

10

e

utility

usability

beauty

stimulation

identity

Strengh of perception

Time of use

Figure 1. Dynamics of each quality dimension (Start and end point of each dimension are based on the mean elic-

ited ratings)

Explication CORPUS asks participants to explicate any change in qual-ity dimensions. To do so, participants told little stories re-lated to or leading to change, so-called change incidents.

Altogether 24 change incidents where gathered. Two broad categories emerged from the data (see Table 2). One group of incidents referred to the familiarity with the product. The other revolved around comparison. Familiarity was further subdivided into "positive", if referring to an improvement, and "negative", if referring to deterioration. Comparison always led to deterioration.

The two groups further differ in their focus, namely the self or others. Positive and negative familiarity is related to the self-oriented process of learning and habituation. Compari-sons, however, more specifically focuses on product user or ownership in relation to other persons.

Table 3 shows the absolute frequency of change incidents in each quality dimension. The gray shade signifies cells with the highest absolute frequency per dimension. Improving usability is in general explained by a growing familiarity with the product. Stimulation was related to negative fa-miliarity (habituation). For beauty and identity all but one change incident was a comparison.

Category Definition Example

Familiarity A process of getting to know the product and getting used to it.

“I needed some time to orient, but after two weeks I knew how everything worked” (positive) “This was really fast, after some months there was nothing new to try” (negative)

Comparison Users compared their mobile phone to an other one (new, friend's phone) and devaluated it

“My boyfriend got a new one, one you can open with a click and it is more round than mine. I’d like this one more”

Table 2: Definitions and statement examples for the two broad categories

There are two interesting points to make. First familiarity is obviously two-edged: It increases usability, while at the same time leading to a strong reduction of stimulation. In both cases the self is the focus. Second, beauty and identity are more related to others, specifically to comparison with other. In the present domain, comparisons are exclusively upward, leading to a deteriorated perception of the own product.

Table 3. Number of reasons/change incidents for the different quality dimensions over time.

CONCLUSION Only few studies take the dynamics of quality perceptions into account and – if so – they tend to focus on usability (pragmatic aspects) solely. From this, one may conclude that HCI regards the product-user-relationship as more or less stable over time. This may not be the case.

We argue for a more dynamic, change-oriented approach to user experience. To this objective, the present study com-bined the idea of pragmatic and hedonic quality dimensions with a focus on how according quality perceptions change over time. The resulting CORPUS interview technique is, thus, specifically designed to track and explain the dynam-

Category Pragmatic Hedonic

UT US S B I

Familiarity (negative)

4 1

Familiarity (positive)

7

Comparison 5 7


77

ics of user experience. In a second step the results could be improved by comparing the results with longitudinal or cross-sectional studies.

Our preliminary results revealed interesting differences in the dynamics of hedonic and pragmatic quality dimensions, which are in line with common sense and model predictions (e.g., derived from [3]). First, pragmatics improved over time, whereas hedonics deteriorated. Interestingly, usability and stimulation were driven by the same, underlying cause, namely familiarity. Familiarity led to a better understanding of the product and, thus, to a better perceived usability. At the same time, however, the product lost its power to sur-prise and stimulate. Note, that this does not necessarily im-ply a causal relation between both dimensions. Very differ-ent features of a product may make it usable or stimulating. Additionally, the exact time course may be of further inter-est. Our crude reconstruction of the shape of change showed stimulation to deteriorate much stronger (and faster) than usability to improve. Future criteria for a "good" interactive product may be rather the way (speed, strength) a quality dimension improves or deteriorates over time, than its absolute value. In other words, a product with a slightly lower initial perceived usability, but a fast and strong "learning curve", may be "better" than a product with a slightly higher initial usability, but no improvement over time at all.

Second, Hassenzahl [3] argued that stimulation is primarily self-related, whereas identity (and beauty) is rather a social phenomenon. Indeed, causes for deteriorating hedonic di-mensions differed accordingly. Whereas causes for a dete-riorating stimulation were self-related (familiarity, habitua-tion), causes for a deteriorating identity and beauty were based on comparisons with others. This supports the as-sumed different nature of stimulation and identifica-tion/beauty.

Interestingly, the dynamics of the global evaluation resem-bled the hedonic dimensions rather than the pragmatic di-mensions. This may lead to the tentative conclusion that the steady deterioration of hedonics (induced by comparisons) is the driving force behind a weakened user-product-relationship. Indeed, Hassenzahl [3] argued that hedonics rather than pragmatics drive bonding to a product (i.e., strong positive relationship), which is supported by the pre-sent findings.

The present study revealed how mobile phones change their character over time: from the more hedonic to the more pragmatic. Other products (e.g., investment goods) may have other dynamics. CORPUS gives the opportunity to gain first – and admittedly primarily exploratory – insights into the way user experience changes over time. This adds an important, but often neglected aspect to the study of user experience.

ACKNOWLEDGMENTS This research is part of the project "Dynamic quality di-mensions in interactive systems" done in collaboration with and supported by Siemens, Corporate Technology (CT IC 7), Munich.

REFERENCES 1. Hartson, H.R., & Castillo, J.C. Remote evaluation for

post-deployment usability improvement. Proceedings of AVI 98, (1998), 22-29.

2. Hassenzahl, M., Platz, A., Burmester, M. & Lehner, K. Hedonic and ergonomic quality aspects determine soft-ware’s Appeal [technical paper]. Proceedings of the CHI 2000 Conference on Human Factors in Computing, New York: ACM (2002), 201-208.

3. Hassenzahl, M. The thing and I: understanding the rela-tionship between user and product. In Funology: From Usability to Enjoyment, M. Blythe, C. Overbeeke, A.F. Monk and P.C. Wright (Eds), (2003), 31-42.

4. Hassenzahl, M., & Tractinsky, N. User experience – a research agenda. Behaviour and Information Technol-ogy, 25(2), (2006), 91-97.

5. Mendoza, V. Novick, D.G. Usability over time, Pro-ceedings of the 23rd annual international conference on Design of communication: documenting & designing for pervasive information, (2005).

6. Prümper, J., Zapf, D., Brodbeck, F. C., & Frese, M. Some surprising differences between novice and expert errors in computerized office work. Behaviour & Infor-mation Technology, 11, (1992), 319-328.

7. Tractinsky, N., Katz, A. S. & Ikar, D. 'What is beautiful is usable', Interacting with computers13 (2), (2000), 127-145.


78

Documents

Dynamics of user experience: How the perceived quality of mobile phones changes over time