Editor: Alberto BroggiUniversity of Pavia, Italybroggi@ce.unipr.it

I n t e l l i g e n t T r a n s p o r t a t i o n S y s t e m s

78 1094-7167/01/$10.00 © 2001 IEEE IEEE INTELLIGENT SYSTEMS

avoidance technologies, enhance the driver’s ability tosense the surrounding environment. Driver assistance andautomation technologies simulate the driver’s sensorimo-tor system to operate a vehicle either temporarily duringemergencies or for prolonged periods.

Such new information and control technologies thatmake vehicles smarter are arriving on the market as op-tional equipment or specialty after-market components.These technologies are being developed and marketed toincrease driver safety, performance, and convenience.However, these disparate components require further sig-nificant integration efforts to create a coherent intelligentvehicle that complements the human driver, fully consid-ering his or her requirements, capabilities, and limitations.

A fully intelligent vehicle must work cooperatively withthe driver.1 New, uncoordinated technologies could deliverexcessive, competing, or contradictory messages anddemands that might distract, confuse, and overwhelm thedriver. This could cause an overload of the driver’s limited

cognitive resources, thereby decreasing the driver’s perfor-mance and safety. An intelligent system senses its environ-ment and acts to reach its objectives. So, its interaction andcommunication channels—that is, its interface—greatlyinfluence the type of intelligence it can display.

Primary requirements for a definition of a coherentintelligent system therefore include intrasystem integra-tion (among all subsystems, including input/output ifapplicable) and intersystems integration (among the sys-tems and environment, including the user if present). Anintegrated, coordinated system is thus a defining feature ofa human-centered intelligent vehicle. Without it, the vehi-cle would simply be a container of potentially overlappingor conflicting technologies.

The quest for human-centered interfacedesign

Recognizing the importance of smart vehicles and thepotential unintended consequences if human factors arenot at the center of their design, the US Department ofTransportation launched the Intelligent Vehicle Initiativein 1997. “The nation that develops and integrates an archi-tecture that provides a seamless interface to the driver willdominate the automobile industry for many years tocome,” stated the US National Science and Technology

ITS technologies provide vehicles with different types

and levels of intelligence to complement the driver.

Information systems expand the driver’s knowledge of

routes and locations. Warning systems, such as collision-

Human-Centered IntelligentVehicles: Toward MultimodalInterface Integration

Massimo Cellario, University of Pavia

As a general rule, the development of innovative systems based on new technological advances cannot be con-sidered complete until they integrate a sufficiently friendly user interface with the average end user. The same con-sideration applies to intelligent transportation systems, and more precisely to intelligent vehicles, where an easy,effective, and immediate form of dialog with the vehicle equipment could make driving safer and more efficient.Many prototypes of intelligent vehicles integrating extremely advanced information and control subsystems havebeen built and proposed, each with its own interface. This installment of the ITS department provides an essentialoverview of the concepts and principles of, and approaches to, the design of interfaces for equipment in futureintelligent vehicles.

If you have any comment on this department, feel free to contact me. I also seek contributions on the currentstatus of ITS projects worldwide as well as ideas on and trends in future transportation systems. Contact me atbroggi@ce.unipr.it; www.ce.unipr.it/broggi.

—Alberto Broggi

Council.2 Among near-term measurementsand standards needs for developing intelli-gent systems, the US National Institute ofStandards and Technology has underlinedthe importance of architecture and interfacestandards to enable adding intelligence totechnology effectively.

The EU as well has identified human-technology interaction (HTI) as a short-term priority for the deployment of roadtransport technology in Europe. TheJapanese government has also providedexcellent organization and funding forsuch standardization.

In this direction, the IT industry hasbegun promoting PC software platforms asa standard, flexible, and integrated solutionfor coordinating in-vehicle information andcontrol technologies. Such technologieswill include systems for security (warning,assistance, and automation), information(navigation assistance and traffic news),communication (Internet access, email, cellphones, faxes, and pagers), and entertain-ment (backseat movies, video games, andnews). Furthermore, personalization, abasic principle in PC technology develop-ment, is strategic from an overall marketperspective, because customization isattractive to both developers and end users.

Interface design issuesThe integration of individual in-vehicle

technologies will be reflected in coordi-nated and streamlined displays and con-trols. Over time, the vehicle will becomeincreasingly sophisticated in how it com-municates information to and accepts com-mands from the driver.

This increasing complexity has under-scored the importance of providing systemdevelopers with human factors guidanceearly during design. Driver-centered design,however, means more than the ergonomicsof “knobs and dials.” It also requires thatdesigners adopt what the Japanese call kan-sei, the infusion of human sensibility.

A key criterion for the development andintroduction of an innovative technology isthat it provides the intended benefits with-out unintended adverse consequences. Dri-ving is potentially dangerous. Although in-vehicle technologies can enhance thedriver’s capabilities and comfort, the dis-tinctive and complex nature of these sys-tems suggests that they could further straindriver capacities and, if not carefully imple-mented, actually exacerbate existing traffic

problems. The design of the driver–vehicleinterface, where the driver interacts physi-cally and cognitively with the vehicle, istherefore critical.

When giving drivers access to such sys-tems inside the vehicle, designers mustconsider not only safety (that is, not over-loading the driver’s information-processingresources) but also driver acceptance andusability.3 Driver acceptance will play acritical role in how intelligent vehicles lookand perform, and the system interface willstrongly influence how a user views andunderstands the system’s functionality.Interfaces must be intelligent, user friendly,effective, and transparent to use.

Careful HTI design can address these

concerns, although many ITS technologieswill likely still require additional drivereducation and training. Accurate HTIdesign lets users easily and naturallydecompose a task into subtasks and mapthem to the system’s functions.4 So, thefirst step in the design process is to outlinethe potential information requirements fora given hypothetical system. This involvesunderstanding the nature of users’ tasks,the ways in which users most naturallydecompose them, and the informationrequired to perform these tasks.

Recent development of intelligent trans-portation systems suggests that drivers willsoon face a mass of new visual, auditory,and tactile information. In an intelligentvehicle, drivers will have access to moreinformation than they are traditionallyaccustomed. So, the vehicle itself will haveto filter information, by selecting andenhancing relevant information only.Therefore, designers will have to make

many decisions concerning what informa-tion to present and how, where, and whento present it.

Information display issues that willdirectly and significantly affect the system’ssafety, usability, and acceptance include

• modality (for example, auditory, visual,and or tactile),

• format (for example, text, map, tone, orvoice),

• location (for example, concentration anddistribution, and head-up or head-down),and

• time (for example, start time, duration,and frequency).

MultimodalityMost research on information displays

has focused on the acceptability of thevisual or auditory modality. Traditionally,drivers depend largely on vision for driving-related information. However, with an in-vehicle information system (IVIS), driversmust perceive many different types of infor-mation, and a system’s exclusive use of onemodality might lead to driver overload.

The lack of coherent and specific guide-lines and standards for multimodal in-vehicledisplays has resulted so far in design by con-sensus, because similar studies often presentcontradictory results or general theoreticalprinciples too difficult to apply directly.

With the overall goal to develop a compre-hensive, usable set of guidelines, the USDepartment of Transportation has recentlybegun conducting simulator experiments thatrelate display modality, format, and locationto additional critical variables such as

• information type, priority, and complexity;• trip status and driving load; and• the driver’s age and subjective workload.

Overall, in virtually every circumstance,carefully designed multimodal displaysappear to be more beneficial than any sin-gle-modality display. Multimodal displayshave exhibited safer driving behavior underevery driving condition, have been moreeffective in route guidance and emergencyresponse, and have exhibited better scoreson many subjective workload measures.5

Generally, the information architectureshould be as simple as possible. If complexinformation is inevitable, a multimodal dis-play will lower the driver’s workload andwill result in better driving performance.

JULY/AUGUST 2001 computer.org/intelligent 79

Drivers will soon face a mass of

new visual, auditory, and tactile

information. In an intelligent

vehicle, drivers will have access

to more information than they are

traditionally accustomed.

80 computer.org/intelligent IEEE INTELLIGENT SYSTEMS

A recent collaboration between the University of Pavia’sComputer Vision Laboratory and the University of Parma’sDepartment of Information Engineering has lead to an inte-grated, multimodal interface prototype developed for theARGO autonomous vehicle. ARGO is a passenger car with areal-time vision-based control system for extracting road andenvironmental information (lane markings, vehicles, andobstacles) and for autonomously steering the vehicle.1

The automatic driving system’s interface (see Figure A) letsthe driver adjust various driving parameters and select one ofthese driving modes:

• Manual: The system monitors the driver’s activity and warnsthe driver of potentially dangerous situations (options: lanedeparture warning and vehicle or obstacle detection).

• Supervised: The system temporarily controls the vehicle tokeep it safe in dangerous situations (options: lane keepingand vehicle or obstacle collision avoidance).

• Automatic: The system controls the vehicle for an extendedperiod (options: lane or vehicle following).

In the first prototype (see Figure A1), adashboard-integrated control panel pro-vided buttons for setting the system’s dri-ving mode and other parameters for thedeveloper. An LED-based display indicatedthe car’s position relative to the lane’s cen-ter. Audio messages warned the driver ofdetected dangerous conditions and con-firmed a command’s reception and execu-tion through vocal messages, simple tones,or special-purpose “earcons” (auditoryicons such as an alarm siren). An onboardvideo monitor functioned as a debuggingtool, providing full visibility of all systemdata as well as vehicle status.

Although this first interface proved effective and simpleenough for anyone to use, the limitations of its hard-codedinteraction paradigm led us to develop a second prototype (seeFigure A2). Although we still principally intend the new inter-face architecture to support the developer (and only partially tosupport an eventual end user), we have designed and imple-mented it as an integrated, coherent, and flexible interaction

framework (see Figure B). Such a framework is suitable for bothautomation system testing and further human-factors experi-ments in driver–vehicle interaction. In particular, we have imple-mented a client-server architecture on ARGO’s core Linux-PCsystem (the server) and on a Windows CE–Pocket PC subsystem(the client)—a personal, configurable, and mobile platform.

The prototype implements a simplified information architec-ture onto a basic and essential multimodal-interaction para-digm: a simple “windows–buttons” metaphor on the PocketPC touch screen lets the driver easily select the system’s drivingmodes and options while providing essential visual and audi-tory control feedback (for example, warning messages andcommand confirmations).

As I previously stated, the PC-based architecture provides apractical and flexible way to coordinate existing in-vehicle infor-mation and control technologies. It also lets us easily expand thesystem’s capabilities uniformly and coherently. Furthermore, thePocket PC’s graphical user interface is an attractive featurefor developers (who can easily redesign the interface and

evaluate different interface languages) andend users (who can adapt the interactiondialogue to their preferences). The device’smobility and extensibility (the ability to inte-grate different functions and programs) offeradditional, distinctive advantages for devel-opers and users.

Future development of ARGO’s systeminterface will focus on advanced human-centered multimodal integration, graduallyshifting the interaction paradigm towardthe average end user. We’ll explore and eval-uate innovative in-vehicle interaction lan-guages and devices, paying particular atten-tion to alternative visual, auditory, andtactile input–output metaphors (for exam-

ple, different graphical solutions, directional sound, steeringwheel feedback, speech recognition, and gaze and gesturerecognition).

Reference

1. A. Broggi et al., Automatic Vehicle Guidance: The Experience ofthe ARGO Autonomous Vehicle, World Scientific, Singapore, 1999.

An Autonomous Vehicle Interface Prototype

Figure A. Two interface prototypes for the ARGO autonomous vehicle: (1) thefirst prototype, a dashboard-integrated control panel; (2) the current prototype,a GUI on a Pocket PC (a sample screen shot showing automatic mode and thelane-following option).

Driver Pocket PC PC Vehicle Infrastucture

User System Environment

Figure B. The driver–vehicle–infrastructure interaction framework (a user–system–environment human–technology interaction scheme) for the second interface prototype (see Figure A2). A satellite handheld device (a Pocket PC) mediatesdriver interaction with the core automation system (a PC); existing additional driver–vehicle relations remain unmodified.

(2)(1)

However, to avoid annoying the user, multi-modal information presentation should beconservative and carefully balanced accord-ing to the capacity and limits of human sen-sory channels. (For example, balancedvisual and auditory information redundancymight reinforce message effectiveness,whereas excessive redundancy might resultin a frustrating and unusable system.) A fullunderstanding of the subtle trade-offs ofmultimodal information display will requiresignificant research.

Additionally, most research on in-vehiclesystems has concentrated on the presenta-tion of navigation and warning information,resulting in accurate and detailed compara-tive literature about past and present inter-face prototypes.6 On the other hand, researchand documentation on automation systemsinterfaces (and on comprehensive multi-function systems integration) is relativelyrare. At the Universities of Pavia andParma, my colleagues and I are developinga prototype for an automation system inter-face as a flexible framework suitable forfurther multimodal ITS functions integra-tion and testing (see the sidebar).

As more applications are developed andintegrated into working systems, researchersshould continue to reevaluate how differentmodalities and modality combinations affectdriver performance.

The vision of a human-centered intelli-gent vehicle is not fixed; it will continu-ously evolve along with technologicalinnovation. A forward-looking approach totechnology, however, will always focus onthe human–machine interface, the mediumthrough which a user communicates with asystem, the point of contact between peo-ple and technology.

References

1. C. Little, “The Intelligent Vehicle Initiative:Advancing ‘Human-Centered’ Smart Vehi-cles,” Public Roads Magazine, vol. 61, no. 2,Sept./Oct. 1997, pp. 18–25; www.tfhrc.gov/pubrds/pr97-10/p18.htm (current 18 July2001).

2. NTSC Human Centered TransportationSafety Team, “The Human Centered Trans-portation System of the Future,” Proc. ITSAmerica 7th Ann. Meeting, ITS America,Washington D.C., 1997.

3. M.C. Hulse et al., Development of HumanFactors Guidelines for Advanced TravelerInformation Systems and Commercial Vehi-cle Operations: Identification of the Strengthsand Weaknesses of Alternative InformationDisplay Formats, tech. report FHWA-RD-96-142, Federal Highway Administration, Wash-ington, D.C., 1998; www.fhwa.dot.gov/tfhrc/safety/pubs/96142/96142.html (current 16July 2001).

4. J. Preece et al., A Guide to Usability: HumanFactors in Computing, Addison-Wesley,Reading, Mass., 1993.

5. Y. Liu and T.A. Dingus, Development ofHuman Factors Guidelines for AdvancedTraveler Information Systems (ATIS) andCommercial Vehicle Operations (CVO):Human Factors Evaluation of the Effective-ness of Multi-Modality Displays in AdvancedTraveler Information Systems, tech. reportFHWA-RD-96-150, Federal Highway Admin-istration, Washington, D.C., 1999; www.fhwa.dot.gov/tfhrc/safety/pubs/96150/96150.html(current 16 July 2001).

6. C. Carney et al., “In-Vehicle Display Iconsand Other Information Elements: LiteratureReview,” tech. report FHWA-RD-98-164,Federal Highway Administration, Washing-ton, D.C., 1998; www.fhwa.dot.gov/tfhrc/safety/pubs/98164/98164.html (current 26July 2001).

JULY/AUGUST 2001 computer.org/intelligent 81

Massimo Cellario is a PhD candidate incomputer science at the Department ofComputer and Systems Engineering of theUniversity of Pavia. His research interestsare multimodal interfaces for human–computer interaction, 3D computer vision,and image synthesis. He received hisDrEng (master’s degree) in informationengineering from the University of Pavia,with a thesis on nonimmersive virtualenvironments. Contact him at the Diparti-mento di Informatica e Sistemistica, Uni-versità di Pavia, via Ferrata 1, 27100Pavia, Italy; cellario@vision.unipv.it;http://vision.unipv.it/people/cellario.

Let your e-mailaddress showyour professionalcommitment.

An IEEE Computer Society

e-mail alias forwards e-mail

to you, even if you change

companies or ISPs.

you@computer.org

The email address of computing professionals