0018-9162/07/$25.00 © 2007 IEEE30 Computer P u b l i s h e d b y t h e I E E E C o m p u t e r S o c i e t y

G U E S T E D I T O R S ’ I N T R O D U C T I O N

C omputing is leading a revolution—everythingwe do is changing at a pace never experiencedbefore in human history. Whether we talkabout the pervasive, ubiquitous, mobile, grid,or even the social computing revolution, we can

be sure that computing is impacting the way we interactwith each other, the way we design and build our homesand cities, the way we learn, the way we communicate,the way we play, the way we work. Simply put, com-puting technologies are increasingly affecting and trans-forming almost every aspect of our daily lives.

Unfortunately, the changes are not always positive,and much of the technology we use is clunky, unfriendly,unnatural, culturally biased, and difficult to use. As aresult, several aspects of daily life are becoming increas-ingly complex and demanding.

We have access to huge amounts of information, muchof which is irrelevant to our own local sociocultural con-text and needs or is inaccessible because it is not availablein our native language, we cannot fully utilize the exist-ing tools to find it, or such tools are inadequate or nonex-istent. Thanks to computing technologies, our optionsfor communicating with others have increased, but thatdoes not necessarily mean that our communications have

Human-centered computing studies the

design, development, and deployment of

mixed-initiative human-computer systems.

HCC is emerging from the convergence of

multiple disciplines that are concerned both

with understanding human beings and with

the design of computational artifacts.

Alejandro Jaimes and Daniel Gatica-PerezIDIAP Research Institute

Nicu SebeUniversity of Amsterdam

Thomas S. HuangUniversity of Illinois at Urbana-Champaign

Human-CenteredComputing:Toward aHumanRevolution

become more efficient. Furthermore, our interactionswith computers remain far from ideal, and too often onlyliterate, educated individuals who invest significantamounts of time in using computers can take directadvantage of what computing technologies have to offer.

GATEWAYS AND BARRIERSWe could argue that knowledge and communications

are two of the main pillars of any society. As informa-tion of all kinds—on products, services, people, maps,and so on—increasingly forms part of the digital ecosys-tem, the computing technologies we develop become,paradoxically, both the gateways to all kinds of resourcesand the barriers to access them. Therefore, in additionto having become critical in improv-ing lives, computing is now essentialto the livelihood not only of the rel-atively few of us who have the nec-essary skills and access to resources,but potentially for everyone.1,2

For the most part, however, thecomputing community designs andimplements computing algorithmsand technologies without fully tak-ing into account our cognitive abili-ties, the ways we perceive and handleinformation, go about our work and life, create andmaintain our social relations, or use our cultural con-text. That is, researchers and engineers often developcomputing technologies in relative isolation.

Most current methodologies start with an idea thatbuilds on or improves existing technologies or thatsolves problems in a particular technological domain,largely ignoring human issues. The obvious outcomesare more powerful, less expensive computers that aremore difficult to use, in some cases even effectivelyslower and less accessible to most of the world’s popu-lation. Given the current rate of penetration of com-puting technologies (directly or indirectly) in almostevery imaginable human activity, it is clear that the exist-ing computing research and development models are nolonger adequate. People who adapt more quickly totechnology (whether the technology is good or bad) havegreater opportunities to benefit at many levels.

Developing technologies in relative human isolationdoes not contribute to alleviating the problems of wealthdistribution, sustainability, and access to healthcare andeducation. Technologies that are difficult to use not onlywaste our time and make our lives worse, they makeaccess to important resources more difficult, particularlyfor people who do not have the education or languageskills of the minority who develop those technologies.

From this perspective, the current path of developmentof computer technologies is not only detrimental but alsodangerous because it contributes to increasing the gapbetween the educated and uneducated, and between the

rich and poor. The problem is even greater if we considerthat the difficulties in using much of the technology avail-able today are leading to a rapidly increasing contentgap—most digital information is produced in developedcountries, from particular cultural perspectives, and inonly a few languages. To make things worse, access tothe information itself is through technology developedwithout considering the local sociocultural context ofthe majority of the world’s population.

Given the developments in recent years in decreasedcomputing costs, increased wireless communications,and the spread of the Internet, the time is ripe for a majorshift in the computing revolution. In our opinion, thegoal of human-centered computing is to focus the com-

puting revolution on addressinghuman abilities and needs.

HCC SCOPEHCC is a set of methodologies that

apply to any field that uses comput-ers in applications in which peopledirectly interact with devices or sys-tems that use computer technologies.The field is emerging from the con-vergence of multiple disciplines thatare concerned both with under-

standing human beings and with the design of compu-tational artifacts. HCC researchers and designers have arange of backgrounds and interests, from computer science, sociology, psychology, and cognitive science toengineering, graphic design, industrial design, and so on.

HCC studies the design, development, and deploymentof mixed-initiative human-computer systems (http://is.arc.nasa.gov/HCC/intro.html). It embodies a systemsview that includes computational tools, cognitive andsocial systems, and physical facilities and environments.HCC “inherits the complexity of software engineeringand systems integration, plus modeling of human-machine and human-human interaction. Advances in theory and modeling require systematic data on suchinteractions in realistically complex environments.” HCCfacilitates the design of effective computer systems thattake into account personal, social, and cultural aspectsand addresses issues such as information design, human-information interaction, human-computer interaction,human-human interaction, and the relationships betweencomputing technology and art, social, and cultural issues.

HCC research has multiple goals and has been car-ried out under this name at several institutions since atleast the early 1990s (www.byte.com/art/9404/sec6/art4.htm). Some researchers have focused on under-standing people, both as individuals and in socialgroups, by studying the ways they adopt and adapt com-putational technologies in their lives, while others havefocused on developing new design strategies for com-putational artifacts (www.human-centered.org).

May 2007 31

HCC embodies a

systems view that includes

computational tools,

cognitive and social systems,

and physical facilities

and environments.

32 Computer

Human-centered design of computational toolsfocuses on problems that traditional human-computerinteraction (HCI) does not generally address. Traditionaldesign approaches often include heuristic evaluationsand measurements of productivity and efficiency. HCCresearchers also bring a diverse array of conceptual andresearch tools to traditional computing areas.

HCC is not just about the interaction, the interface, orthe design process. It is about knowledge, people, tech-nology, and everything that ties them together. Thisincludes how the technology is actually used and in whatcontext. Furthermore, HCC involves both the creationof theoretical frameworks and the design and imple-mentation of technical approaches and systems in manyareas (www.nsf.gov/cise/iis/about.jsp).

Both HCC’s scope and its possibilities are wide, butin our view, three factors form the core of HCC systemand algorithm design processes:

• augmenting or taking intoaccount individual human abili-ties and limitations,

• social and cultural awareness,and

• adaptability across individualsand specific situations.

If we consider these factors in thedesign of systems and algorithms,HCC applications should exhibit the following qualities:

• integrate input from different types of sensors andcommunicate through a combination of media asoutput,

• act according to the social and cultural context inwhich they are deployed, and

• be useful to diverse individuals in their daily life.

In considering the scope, application areas, and qual-ities of HCC systems, much can be learned from fieldsand disciplines related to HCC.

HCC, HCI, AND HUMAN COMPUTATION Many important contributions to HCC have been

made in fields such as HCI, computer-supported coop-erative work (CSCW), user-centered design,3 cognitivepsychology, sociology, anthropology, and others.

User-centered design can be characterized as a multi-stage problem-solving process that requires designersnot only to analyze and foresee how users are likely touse an interface but also to test the validity of theirassumptions with regard to user behavior in the realworld.4 CSCW combines the understanding of the waypeople work in groups with the enabling technologiesof computer networking and associated hardware, soft-ware, services, and techniques.5 In contrast, HCC

covers more than the traditional areas of usability engi-neering, HCI, and human factors, which are primarilyconcerned with user interfaces or user interaction.6

Compared to HCI, HCC has a twofold perspective(www.cs.berkeley.edu/~jfc/hcc):

• HCC is “conceived as a theme that is important forall computer-related research, not as a field that over-laps or is a subdiscipline of computer science”; and

• HCC acknowledges that “computing connotes bothconcrete technologies (that facilitate various tasks)and a major social and economic force.”

Interactive evolutionary computation, an interestingtechnique first developed in the early 1990s and morerecently known as human computation,7 also puts thehuman at the center, but in a different way. In traditional

computation, a human provides aformalized problem description to acomputer and receives a solution tointerpret. In human computation, theroles are reversed: The computer asksa person or a large number of peopleto solve a problem, then collects,interprets, and integrates their solu-tions. In other words, the computer“asks” the human to do the work itcannot do. This is done as a way toovercome technical difficulties:

Instead of trying to get computers to solve problems thatare too complex, use human beings.

In human computation, the human is helping the com-puter solve difficult problems, but in HCC, the com-puter helps humans maximize their abilities regardlessof the situation—the human need not be in front of acomputer performing a computing task when dealingwith the computer.

Although HCC and human-based computationapproach computing from two different perspectives,they both try to maximize the synergy between humanabilities and computing resources. Work in human com-putation can therefore be of significant importance toHCC. On one hand, data collected through human com-putation systems can be valuable for developingmachine-learning models. On the other hand, it can helpus better understand human behavior and abilities,again of direct use in HCC algorithm development andsystem design.

RESEARCH AGENDA AND KEY HCC CHALLENGES HCC systems should be multimodal—processing

inputs and outputs in a naturally rich communicationchannel; they must be proactive—understanding cul-tural and social contexts and responding accordingly;and they must be easily accessible outside the desktop toa wide range of users.8

HCC researchers

bring a diverse array

of conceptual

and research tools

to traditional

computing areas.

A human-centered approach to computing thus needsto consider how human beings understand and interpretmultimedia signals at the perceptual, cognitive, andaffective levels, and how we interact naturally—embed-ding the cultural and social contexts as well as personalfactors such as emotion, attitude, and attention. Thisrequires considering work in fields such as neuroscience,psychology, cognitive science, and others. A key chal-lenge is incorporating what is known in those fieldswithin computational frameworks that integrate differ-ent media.

Research on machine learning integrated with domainknowledge, data mining, sensor fusion, and multimodalinteraction will play a key role.9 Further research intoquantifying human-related knowledge is necessary,which means developing new theories and mathemati-cal models of multimedia integration at multiple levels.We believe that an HCC researchagenda will involve a nonexhaustivelist of goals including the following:6

• new human-centered method-ologies for the design of modelsand algorithms and the develop-ment of diverse HCC systems;

• focused research on the integra-tion of multiple sensors, media,and human sciences that havepeople as the central point;

• new interdisciplinary academic and industrial pro-grams, initiatives, and meeting opportunities;

• discussions on the impact of computing technologythat include the social, economic, and cultural con-texts in which such technology is or might bedeployed;

• research data that reflects human-centeredapproaches—for example, rich data collected fromreal multisensorial and culturally diverse social situ-ations;

• common computing resources—for example, soft-ware tools and platforms;

• evaluation metrics for theories, design processes,implementations, and systems from a human-centeredperspective; and

• methodologies for privacy protection and the con-sideration of ethical and cultural issues.

Clearly, an HCC research agenda should include abroad understanding and a multidisciplinary approach,as Eric Brewer and colleagues proposed in the specificcontext of developing regions.1

IN THIS ISSUEThe call for papers for this special issue prompted an

extraordinary response from the computing community.We received 130 submissions, which in our opinion

reflects the increasing interest in HCC. The number ofsubmissions, as well as the scope of the work submit-ted, is also indicative of the challenges in HCC.Although research is being carried out in many areas,much work needs to be done to define HCC’s theoreti-cal foundations and to gain a clearer understanding ofwhat constitutes an HCC methodology or application.One goal of this special issue and of the newly createdIEEE Computer Society Task Force on Human-CenteredComputing is to restart the discussions on the founda-tional aspects of HCC (www.human-centered.org).

For this special issue, we selected four articles that coverapplication areas of interest to HCC researchers and prac-titioners. The articles exemplify some of the importantprinciples in human-centered computing and representsome current trends and developments in the field.

In “A Communication Support System for OlderPeople with Dementia,” NormanAlm and colleagues present a com-puter-based communication supportsystem that demonstrates how sim-ple technology can be used effec-tively to assist older people withdementia in carrying out conversa-tions. The work presented in thisarticle is truly a multidisciplinaryHCC effort developed by partici-pants from software engineering,psychology, and design. The system

provides cognitive communication support via a richhypermedia display of material from the past that facil-itates communication between patients, caretakers, andfamily members.

“A Language for Human Action” by GutembergGuerra-Filho and Yiannis Aloimonos advocates the ideathat the space of individual human actions can be seenas having a structure similar to that of language. Theauthors introduce a human activity language (HAL) thatrepresents visual and motor information via symbols.They address three issues: phonology, where they define“atomic” segments used to compose human activities;morphology, which determines the structure of eachaction and their organization in an action lexicon; andsyntax, which incorporates basic constraints to formu-late action sentences. A language for human action pro-vides an intriguing link between human language andhuman motion, and the authors have used the princi-ples behind it to build a computational framework.

In “An Interactive Multimedia Diary for the Home,”Gamhewage C. de Silva, Toshihiko Yamasaki, andKiyoharu Aizawa present a video retrieval and summa-rization system for a home environment equipped with alarge number of sensors. In this application, the computeris actually part of the environment and the focus is onrecording daily human activity, not interaction with anyparticular computing device. In addition to analyzing

May 2007 33

HCC systems

should be multimodal,

proactive, and

easily accessible

to a wide range

of users.

34 Computer

the signals obtained from pressure-based sensors mountedin the floor, the system uses audio and video data streamsto detect simple events. The ultimate goal is to incorpo-rate multiple channels of information to allow detectionand recognition of high-level actions and events such asconversations and to facilitate the creation of interactivemultimedia diaries for life-log applications.

Finally, “Holistic Sensing and Active Displays forIntelligent Driver Support Systems” by Mohan M.Trivedi and Shinko Y. Cheng presents an overview ofmultidisciplinary research on the design and evaluationof computational systems for improving driving safety,focusing on a novel dynamic active display. The authorsalso discuss the design of novel instrumented vehiclesused for conducting realistic driving experiments inwhich researchers record information about the environment, vehicle, and driver state, both for ethno-graphic studies and for signal analysis geared towardimproving driving safety.

W e hope the readers of this special issue find thesearticles exciting, as we believe they could lead to restarting discussions on human-centered

computing. ■

Acknowledgments

We thank all of the authors who submitted articles forthis special issue. Given the number of submissions, thereview process involved two stages. We first preselected40 submissions based on their overall quality and rele-vance to the call for papers. Each of these papers wasthen reviewed by at least two independent reviewers.The acceptance decision involved multiple discussionsbetween the special issue editors and Scott Hamilton,Computer’s senior acquisitions editor. We also thankNahum Gershon for fruitful discussions on this topic.

References

1. E. Brewer et al., “The Case for Technology in DevelopingRegions,” Computer, June 2005, pp. 25-38.

2. R. Jain, “Folk Computing,” Comm. ACM, vol. 46, no. 4,2003, pp. 27-29.

3. J. Karat and C.M. Karat, “The Evolution of User-CenteredFocus in the Human-Computer Interaction Field,” IBM Sys-tems J., vol. 42, no. 4, 2003, pp. 532-541.

4. D.A. Norman and S.W. Draper, User-Centered System Design:New Perspectives on Human-Computer Interaction,Lawrence Erlbaum, 1986.

5. J. Grudin, “Computer-Supported Cooperative Work: Its His-tory and Participation,” Computer, May 1994, pp. 19-26.

6. A. Jaimes, N. Sebe, and D. Gatica-Perez, “Human-CenteredComputing: A Multimedia Perspective,” Proc. 14th Ann.ACM Int’l Conf. Multimedia, ACM Press, 2006, pp. 855-864.

7. C. Gentry, Z. Ramzan, and S. Stubblebine, “Secure Distrib-uted Human Computation,” Proc. ACM Conf. ElectronicCommerce, LNCS 3570, Springer, 2005, pp. 328-332.

8. A. Jaimes, “Human-Centered Multimedia: Culture, Deploy-ment, and Access,” IEEE MultiMedia, vol. 13, no. 1, 2006,pp. 12-19.

9. A. Pentland, “Socially Aware Computation and Communi-cation,” Computer, Mar. 2005, pp. 33-40.

Alejandro Jaimes is scientific manager and a seniorresearcher at IDIAP Research Institute, Martigny, Switzer-land. His research focuses on developing computer visiontechniques that use machine learning, involve humansdirectly, and are rooted in principles, theories, or techniquesfrom cognitive psychology, the arts, and information sci-ences. Jaimes received a PhD in electrical engineering fromColumbia University. Contact him at alex.jaimes@idiap.ch.

Daniel Gatica-Perez is a senior researcher at IDIAPResearch Institute. His research focuses on automatic analy-sis of human activities and social interaction from sensordata, integrating concepts from multimedia signal process-ing, machine learning, and social psychology. Gatica-Perezreceived a PhD in electrical engineering from the Univer-sity of Washington. Contact him at gatica@idiap.ch.

Nicu Sebe is a professor in the Department of ComputerScience at the University of Amsterdam, The Netherlands.His research interests include human-computer interactionfrom a computer vision perspective and multimedia infor-mation retrieval. Sebe received a PhD in computer sciencefrom Leiden University, The Netherlands. Contact him atnicu@science.uva.nl.

Thomas S. Huang is the William L. Everitt DistinguishedProfessor of Electrical and Computer Engineering, aresearch professor at the Coordinated Science Laboratory,and head of the Image Formation and Processing Group atthe Beckman Institute for Advanced Science and Technol-ogy, University of Illinois at Urbana-Champaign. Hisresearch interests focus on the broad area of informationtechnology, especially the transmission and processing ofmultidimensional signals. Huang received a PhD in electri-cal engineering from MIT. He is a member of the NationalAcademy of Engineering, a Foreign Member of the ChineseAcademies of Engineering and Sciences, and a Fellow of theInternational Association of Pattern Recognition, the Opti-cal Society of America, and the IEEE. Contact him athuang@ifp.uiuc.edu.