30 March 2003/Vol. 46, No. 3 COMMUNICATIONS OF THE ACM

Attentive User Interfaces

If there was a Moore’s Law for user interfaces, it would state that the number of computers per user will double every two years. In the past fourdecades, we have moved from many users sharing a single mainframe computerthrough command line interfaces, to a single user with a personal computer usinga graphical user interface (GUI). Today,increasing numbers of users are surroundedby multiple ubiquitous computing devices,such as BlackBerries, PDAs, and cell phones. As our devices connect to a globalwireless network, we become members of a 24-hour global society—one wherewe are always connected, and always on.

� By Roel Vertegaal, Guest Editor

COMMUNICATIONS OF THE ACM March 2003/Vol. 46, No. 3 31

Although the trend to use more computingdevices may provide an opportunity forincreased productivity, such benefit comes at acost. That cost is the requirement to be avail-able—at any time or place—in order to swiftlyadapt to changes in our information environ-ment. Rather than mitigate this cost, our com-puting devices currently exacerbate it. This isbecause their user interfaces have not funda-mentally changed in 20 years: each device acts inisolation, as if it were the user’s only device. As aconsequence, devices bombard users withrequests for attention, regardless of the cost oftheir interruptions. At the same time, comput-ers are increasingly enveloping our senses by

becoming part of our physical environment.Although ubiquitous wall-sized displays mayoffer relief from the tunnel vision caused by ourdesktop screens, this technology brings with itthe capacity to further increase our informationload. Consequently, growing demands on users’attention have become a critical usability issuein computing.

Researchers are becoming aware of the factthat user attention is a limited resource thatmust be conserved. User interface designers andengineers are beginning to design computingdevices that negotiate rather than impose thevolume and timing of their communicationswith the user. Devices are augmented such that

Illustration by Jean François Podevin

32 March 2003/Vol. 46, No. 3 COMMUNICATIONS OF THE ACM

they become aware of competing requests for theiruser’s attention. Information systems are beingdeveloped that weigh the importance of the infor-mation they supply with the estimated priorities inuser activity. Displays are created that track the user’sfocus of attention and adjust their renderings to pro-vide peripheral context in support of focused activ-ity. This area of work has created new interactionand visualization styles that aim to focus—ratherthan distract—the user. Computing interfaces thatare sensitive to the user’s attention are called Atten-tive User Interfaces (AUIs); the user interface para-digm that is the topic of this special section.

AUIs follow the general metaphor of a well-tunedmodern traffic light system. Modern systems usesensors in the road to determine where users will gonext—their future focus. They employ statisticalmodels of traffic volume to determine the priority ofuser requests for intersection space. They useperipheral displays—traffic lights—to negotiateturn-taking activities of users sharing the limited realestate of an intersection. Likewise, AUIs may mea-sure and model the focus and priorities of their user’sattention. They structure their communication suchthat the limited resource of user attention is allo-cated optimally across the user’s tasks.

AUIs use a range of novel input, interpretation,and output techniques to implement the examplesmentioned here. To determine what task, device, orperson a user is currently attending to, AUIs employextra channels of input that measure overt charac-teristics of user attention (such as user presence,proximity, orientation, speech activity, or gaze).Based on such input, AUIs infer knowledge aboutthe priorities in the tasks that govern their user’sattention. By statistically modeling attention andother interactive behaviors of users, AUIs may estab-lish the urgency and relevance of information oractions they offer in the context of current activity.Such interfaces with a deep understanding of user

attention have been referred to as Attentional UserInterfaces. We note that models of the attentivebehavior of users need not be fully predictive. Thisis because AUIs are able to gracefully negotiate theamount of attention they receive from the user usinga process similar to turn-taking behavior in conver-sations. Instead of immediately taking the fore-ground—interrupting the ongoing activity of theuser—AUIs can progressively signal their requests forattention. Initially this may happen through a chan-nel peripheral to the user’s activity. AUIs may thenwait for user acknowledgment—provided throughan attentive input device—before they take the fore-ground. To achieve such behavior during mediatedcommunications, AUIs may relay information aboutthe attention of their users to remote persons ordevices. Such interfaces are known as “Attention-based” or “Awareness” systems.

Finally, AUIs can also augment the user’s capacityfor attention. Analogous to the Cocktail PartyEffect, which allows people to focus on one amongseveral speakers during noisy meetings, AUIs canenhance information presented in areas where usersfocus their attention, while they attenuate peripheraldetail. The accompanying figure summarizes thefeatures noted here, and compares them to interac-tion techniques currently used in GUIs.

The focus on design for user attention is what dis-tinguishes the AUI paradigm from other, relatedareas of research in human-computer interaction.For example, Perceptual User Interfaces aim to opti-mize human-computer interactions by employingusers’ verbal and nonverbal communication chan-nels. AUIs employ such perceptual information witha specific intent to identify or optimize user atten-tion. Similarly, Context-Aware Interfaces use thecontext in which interactions take place to providemore relevant services to the user. However, Con-text-Aware Interfaces are attentive only when thatcontext is governed by user attention. The AUI par-

Devices bombard users with requests for attention, regardless of the

cost of their interruptions.

COMMUNICATIONS OF THE ACM March 2003/Vol. 46, No. 3 33

adigm also has roots in a history of research onhuman attention, eye tracking, nonverbal communi-cation, conversational interfaces, user modeling, andawareness in collaborative systems. The phrase“attentive user interface” is used in this section as anumbrella term that includes attentional and atten-tion-based systems, as well as what have been called“attentive systems,” which observe user activity toanticipate user needs.

As with every emerging paradigm, AUIs bringabout new challenges for researchers. The articles inthis section present some of the first milestonestoward Attentive User Interfaces. Authors contribut-ing to the section include some of the world expertson this topic who share their views and present theirprototypes. Zhai lays the foundation by taking an in-depth look at IBM’s work on eye tracking—the mostrelevant attentive input technique. He discusses howimplicit measures of the user’s visual attention maycombine with manual action to optimize GUI inter-actions. Shell et al. discuss work at Queen’s Univer-sity and MIT toward AUIs that employ subtleturn-taking techniques to optimize user interactionswith groups of computing devices. Maglio andCampbell describe their research on attentive agents,

information systems that track user interest and ren-der suggestions using peripheral ticker-tape displays.Horvitz et al. review principles for sensing and rea-soning about a user’s attention with Bayesian mod-els, and for using these inferences to identify idealdecisions in messaging, interaction, and computa-tion. Baudisch et al. take a look at advances in atten-tive display technologies, visualization systems thatincrease or decrease information density dependingon the focus of user attention. McCrickard andChewar close this section by proposing a qualitativeframework for evaluating AUI designs.

Roel Vertegaal (roel@acm.org) is a professor of human-computer interaction and director of the Human Media Lab atQueen’s University, Kingston, Ontario, Canada.

Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. To copy otherwise, to republish, to post on servers or to redistributeto lists, requires prior specific permission and/or a fee.

© 2003 ACM 0002-0782/03/0300 $5.00

c

1960s–1980smany-one

1980s–1990sone-one

1990s–2000sone-many

2000s–2010smany-many

alert

Undo

Cut

Copy

Paste

Select all

Clear

This diagram shows how AUI interaction styles relate to those found in traditional GUIs. AUIs sense whether they are in the foreground

of user attention by observing the presence, activity, and

eye-gaze behavior of the user. In AUIs, menus and alerts

are supplanted by a gracefulnegotiation of turns with the

user through peripheral channels. Foreground

activities are distinguished from background activities using seamless transitions

in the density of the information rendered,

rather than by a discrete windowing system.

eye contact turn interruption high-detail focus vs. low-detail background

pointer menu alert focus window vs. background icon

AUI

GUI