14 1541-1672/06/$20.00 © 2006 IEEE IEEE INTELLIGENT SYSTEMSPublished by the IEEE Computer Society

I n t e r a c t i v e E n t e r t a i n m e n t

Intelligent InteractiveEntertainment Grand ChallengesMark Maybury, MITRE

Oliviero Stock, Trentino Cultural Institute Center for Scientific and Technological Research

Wolfgang Wahlster, German Research Center for Artificial Intelligence

W ebster’s defines entertainment as “something diverting or engaging” or “a per-

formance.” IT occurs in a broad range of domains including games and toys,

the fine arts, movies and radio, sports, travel, and education. Figure 1 shows some exam-

ples from the digital world, including virtual games, interactive arts such as audience-

guided movies, augmented sports (performance orviewing), virtual and nonvirtual travel guides, andcomputer-based tutors. In each case, there are vari-ous ways to support users—for example, by guidingthe material and information, by enhancing the users’performance, by expertly critiquing their perfor-mance, or by providing online help or guidance.However, providing support requires knowledge notonly about the particular entertainment domain (andits associated entities, relationships, tasks, and activ-ities) but also about interaction competence.

Intelligence in gamesAI encompasses multiple facilities:

• Perception includes seeing, hearing, and feeling.• Cognition addresses analysis, planning, and

forecasting.• Communication embraces speech, language, ges-

ture, and discourse.• Manipulation includes creation, movement, and

destruction of entities.• User and agent modeling enables representation

and reasoning about beliefs, preferences, goals,and plans.

• Learning addresses abstraction, generalization,failure analysis, and adaptation to novel situations.

Each area offers valuable capabilities to digital enter-tainment. Figure 2 considers how these facilitiesapply to intelligent games. For example, perceptual

processing such as highlighting objects or events ofinterest can enhance user experience and memory.The enhanced realism of intelligent behavior or strat-egy representations can enrich cognition. Human-language technologies promise more effective com-munication between users and the game as well asmore natural and believable game-character com-munication. Kinesthetic models can more effectivelyrepresent the virtual world and improve the abilityto reason about and manipulate embodied charac-ters. Agent languages support the design and rea-soning of more natural artificial characters. Finally,user models enable the personalization of informa-tion and presentations to a changing set of user goals,beliefs, and preferences.1

Intelligent interactionHuman-computer interaction is an AI application

area that addresses aspects of an intelligent interface:As figure 3 shows, these aspects include multimodal,tailored, interactive, cooperative, mixed-initiative,and model-based interactivity.

Multimodal interfaces support multimedia (forexample, spoken or written language, gesture, gaze,and graphics) and multimodal input analysis and out-put generation (for example, audio, visual, and hap-tic) as well as multimodal discourse processing (forexample, within and across modal reference and dis-course). Cooperative interfaces help users detect andcorrect misattributions or misconceptions, avoidundesirable events and states, and complete dele-

Advances in AI and

human-computer

interaction offer

seemingly endless

opportunity to enhance

traditional forms

of entertainment and

support the creation

of new ones. However,

several fundamental

scientific and technical

impediments must be

overcome to achieve

these promises.

gated tasks. Mixed-initiative interfaces letthe user or system at any point assume theleading role in the interaction; they requirethe flexibility to lead a dialogue and respondnaturally when a user barges in. Tailoredinteraction utilizes task, user, or discoursemodels to customize the interface; the mod-els are based, for example, on task needs oruser preferences. Model-based interfacesincorporate declarative models of the user,domain, task, style guides, and interface ele-ments to support the semiautomated designand redesign of the interface itself.

Intelligent interfaces can support entertain-ment users in various roles, as illustrated at theright in figure 3. They can guide users throughentertainment materials and activities, serve asmentors or tutors to help users perform activ-ities, act as expert critics of user plans ordesigns, and actually enhance individual orteam performance. This interaction couldrange from a simple direct-manipulationinterface (for example, a recommended menuof actions) to an embodied conversationalagent. Embodied characters afford manyinstantiations, such as a virtual mentor char-acter in an interactive game, an augmented-reality tour guide in an art museum, coplay-ers in a virtual orchestra, or a personal tutorin a virtual classroom.

Domain challengesGiven the role of intelligence in entertain-

ment generally and in the interface in partic-ular, we next consider possible challenges ineach entertainment domain.

Interactive gamesJust as humans compete in Olympic games

that include both individual events anddecathlons that assess overall athletic achieve-ment, digital interactive games encompassboth individual challenges and ones that drawupon multiple intelligences.

Interactive, intelligent games have a longhistory, including specialized expert gameplayers such as IBM’s Deep Blue, which beatworld chess champion Gerry Kasparov in1997. After that controversial success, basedon special hardware, the field continued toadvance by extending the use of AI. Currentleading chess programs are probably strongerthan Deep Blue, even if they run on normalPCs. Researchers are also developing manyother game experts, including a system thatcan beat humans in solving real crosswordpuzzles.2 This system requires specific nat-ural-language and information-retrievalcapabilities.

Besides specialized programs, AI isaddressing general game playing. In GGP,programs must learn rules about games asopposed to just perform within a singlegame’s predefined rule set. The 2005 AAAIGGP competition aimed to “test the abilities

of general game playing systems by compar-ing their performance on a variety of games.”3

Players are told nothing about the games to beplayed before the competition. At the begin-ning of each game, they receive its rules elec-tronically in the GGP Game Description Lan-guage (http://games.stanford.edu/language.html). Systems compete against one anotherfor a US$10,000 prize.

Measuring a game’s learning competenceis one challenge. Assessing its enjoyability orrealism from the user’s perspective poses otherchallenges. These might include creatinggames that let the user and system collaborateto predict some quasi real-world outcome,such as the price of a barrel of oil in a Sims-like world or the most effective immigrationpolicy. These games might measure successby how fast the system-user team achieved thegoal or by their proximity to the optimal solu-tion within some specified time period.

Interactive digital artsFigure 4 illustrates a range of ways in

which intelligence and interaction can en-hance the digital arts.

SEPTEMBER/OCTOBER 2006 www.computer.org/intelligent 15

Digitalentertainment

Virtualgames

Interactive music

Interactivemovies

Augmented sports

Entertainmentdomains

Games andtoys

Fine arts

Sports

Travel

Education

Virtualtour guides

Cinema, TV,and radio

Computer-basedtutors

Figure 1. Digital entertainment domainsand examples.

Intelligent-game features

Augmented perception(for example, object highlighting and camera control)

Character behaviors, game strategies

Human-language technology for characters,natural user input

Environment modeling, embodied characters

Artificial characters

Personalization

AI

Perception

Cognition

Communication

Manipulation

Artificial agents

User models

Figure 2. AI facilities featured in intelligent games.

AI Intelligent interaction

Perception

Cognition

Communication

Manipulation

Artificial agents

Multimodal

User models

Tailored

Cooperative

Model based

Mixed initiative

Tutors,mentors

Expert critics

Guides

Performanceenhancers

User support roles

Figure 3. Intelligent interaction supports entertainment users in many roles

Visual, acoustic, or haptic devices can aug-ment both perception and performance. Suchdevices could facilitate performance for peo-ple who are perceptually challenged or justmake it easier for average users. A primitiveexample is visual highlighting of the keys oninteractive-training keyboards. An acousticexample is automatic pitch detection and cor-rection in Karaoke machines to assist tone-deaf or off-key singers by readjusting themusic pitch to match the singer.

Graphic design, music composition, anddance choreography all require detailedknowledge of presentation elements, theirproperties, the events that prompt them, andhow to create those events. Performance crit-ics and mentors can use this information to sup-port learning and performance. Researchershave already demonstrated content-basedmultimedia search to discover artifacts suchas pictures, sounds, and movements.4

Beyond simple search, embodied conver-sational guides could provide access to largeartifact collections such as those in museumsor museum archives. We could use modelsof individual interests and preferences to per-

sonalize these guides in terms of content,sequence, and presentation.

Embodied agents could serve as partnersfor dance or musical performances. Popularstatic forms of this idea already exist, such asthe Music Minus One (www.musicminusone.com) series. A more interactive, reactive formcould include virtual musicians or dancersthat would react to a user’s actions and pos-sibly improvise their responses.

One challenge would be to have a noviceartisan perform with assistance (either aloneor with other artisans). A panel of humanjudges could then assess the quality and styleof the resultant performance, which mightfeature drawing, painting, dancing, or play-ing an instrument. In another form, the inter-activity could support coordination amongvarious artists—for example, choreographyfor orchestral or band song composition. Achallenge task might be to support noveldesigns or competition against an experthuman (a kind of artistic Turing test).

Interactive cinema, TV, and radioFigure 5 illustrates a range of intelligence

applications to digital cinema, TV, and radio.Directors often enhance movies during pro-duction by highlighting particular objects orevents (for example, the red-dressed girl inthe black-and-white movie Schindler’s List).An intelligent design critic can comment onmovie scripts or storyboards. By designingchoices in plot lines, producers can give view-ers an active role in a story’s progression,although outcomes are generally predefined.Human-language technology or manipula-tion-enhancement methods can provide nat-ural multimodal access to cinema, TV, andradio for all. These artifacts are challengingbecause they contain mixes of speech, music,and natural sounds. Directors and producerscommonly use nonlinear digital editors today.Increasingly, tools incorporate content-basedsearch, which can include model-based under-standing of broadcast news,5 movies, andvideotaped meetings. User models alreadysupport recommender systems for movies,books, and music.

Example interactive-movie challenge tasksinclude finding key music or video segmentsor answering questions about characters’behaviors or events in a movie. We couldmeasure performance in terms of both taskaccuracy and timeliness or perhaps even userenjoyment.

Interactive digital sportsLike many forms of entertainment, users

can enjoy sports both as performers andobservers. Sports observation is typically inperson or via television. The examples fromthe previous section are relevant in the caseof TV. In the case of performance, the imple-mentations can be either physical or virtual.Virtual interactive games are very popular inmany sports such as football, baseball, soc-cer, and hockey. Users can play as individu-als or team managers, participating directlyin play and strategy; they can also modifyplayer parameters, select situations, and evenmanage camera control. AI plays a key rolein digital sports to include player, team, andcrowd behaviors; human-language technol-ogy for character interaction; as well as aug-mented perception by manipulating cameraangles and replay speeds to achieve enter-taining effects.

In addition to virtual sports, a variety ofaugmented-reality sports have become pop-ular. Examples include paintball and lasertag. These sports combine sensors and effec-tors with computers to, for example, countthe number of hits or dynamically establish

I n t e r a c t i v e E n t e r t a i n m e n t

16 www.computer.org/intelligent IEEE INTELLIGENT SYSTEMS

Digital cinema, TV, and radio

Augmented perception and performance(for example, object highlighting and camera control)

Digital critics, mentors

Human-language technology for access, artificial guides

Embodied agents, augmented manipulation

Interactive movies, movie planning

Personalization(for example, TV guides and recommenders)

(character behavior, character movie background)

AI

Perception

Cognition

Communication

Manipulation

Artificial agents

User models

Figure 5. Challenges in the digital cinema, TV, and radio domain.

Digital arts (for example, painting, dance, and music)

Augmented perception and performance(for example, object enhancement and pitch detection)

Digital critics, mentors, and search

Artificial guides (museums, music libraries)

Augmented manipulation (training, assisting impaired people)

Virtual musicians (for example, Music Minus One)

Personalization

(choreography, composition, content-based multimodal search)

AI

Perception

Cognition

Communication

Manipulation

Artificial agents

User models

Figure 4. Interactive-entertainment challenges in the digital arts.

team memberships, thus enhancing the situ-ations and reactions to situations that play-ers experience individually and collectively.

Interactive sport challenges arise naturallyin the games themselves, of course. Becausegames can measure both individual and teamperformance, they provide an interesting setof possible success measures. These caninclude not only raw scores or hit/miss ratiosand more fine-grained individual-skill assess-ments (such as posture and movement in timeand space) but also the quality of strategy orthe degree of distribution or success in coor-dinated team activity.

Interactive travelVirtual or augmented travel has become a

popular preparation for or alternative to phys-ical travel. Virtual tour guides similar to thoseused in museums can help plan any itinerary.They might interactively tailor itinerariesaccording to the interests of a particular trav-eler or travel group. They might also supporthuman-language interaction by answeringquestions about historic sites. These mighteventually be mobile robots, but they couldjust as well be disembodied agents that inter-act with users via handheld mobile platforms.In Boston, visitors to Zoo New England (theFranklin Park or Stone Zoos), Minute ManNational Historical Park, or the city canreceive information about current attractionsinteractively via their cell phones in their pre-ferred language and voice, using technologydeveloped by Spatial Adventures (www.spatialadventures.com).

Intelligent agents might do much more insupport of travel, such as planning itinerariesoptimal to user’s desiderata, scheduling routessensitive to local weather conditions, or sim-ply negotiating transportation purchases. GPSand enabled wireless devices with onboarddigital maps (including 3D and subsurfaceapplications) enable diverse possibilities.

An interactive travel challenge could mea-sure performance of an intelligent agent oragent-supported user to perform a travel-planning task within a given time to maxi-mize some set of measures such as a combi-nation of economy, excitement, and novelty.Or you could simply have a scavenger huntdriven by interesting locations.

Interactive educationSo-called “edutainment” already incorpo-

rates interesting interactive elements. Reading,writing, and drawing assistants help noviceslearn basic skills. Some incorporate, for exam-

ple, speech processing (including accent mod-els) so that young readers learning their nativeor a foreign language receive immediate andindividualized performance feedback. Incor-porating interactive games seems to enhanceuser motivation. In the future, incorporatingquestion-answering systems could help pro-vide more tailored mentoring.6,7

Animated pedagogical agents in interac-tive learning environments enable effectivesituated learning.8 For example, researchershave shown that a qualitative simulation ofthe cardiac cycle in a virtual patient providesa safe, effective 3D virtual emergency-roomtraining environment for student physicians.9

Challenges are evident in tasks such astraining and testing users both with and with-

out intelligent interactive training. Metrics forevaluating this work include error rates andtime to learn. A problem’s complexity levelor team-based problem-solving could alsoestablish interesting evaluation opportunities.

Home and society challengesWhile perhaps not obvious as an enter-

tainment form, home and other social envi-ronments can be natural spaces for interac-tive entertainment. For example, some peoplecook for fun; an interactive assistant coulddo everything from help with online shop-ping and menu planning to suggesting musicor games to play at a dinner party. Or it couldhave specialized knowledge for home repairor gardening. Interactive pets can providecompanionship. A finding agent could keeptrack of a child’s whereabouts or more mun-dane things such as the remote control or alost sock. A social-analysis agent might helpdiscover social opportunities based on social-network analysis or simply remind the userof important dates such as birthdays andanniversaries.10

Interactive-home challenges range from

keeping a house clean to advising on keytasks. Measuring success in this environmentincludes the number of errors, cost savings,and increased user satisfaction.

Communication as entertainment

Besides its applications in artistic forms,communication in general can be an enter-taining activity. When artificial agents becomepart of our environment, we expect theircommunication to have the same basic prop-erties as human communication. For exam-ple, they might exhibit humor, storytellingcapabilities, or even produce pictures orvideo clips to share important information.

The entertaining properties of communi-cation can play a practical role in many situ-ations. Consider humor in advertising, whereit first helps get the audience’s attention andthen helps them to remember the message.Or think of the essential role of entertainmentin children’s education.

If a storytelling system can engage the lis-tener, for example, it can provide a goodcompanion to a senior person and convinceher of the importance of taking medicine anddoing some exercise. Such a system can alsoprovide live sport commentary.

Entertaining communication is beginningto deliver some initial prototypes. Humor isa complex capability to reproduce, and someexperts have considered it to be “AI com-plete.” It’s nevertheless possible to modelsome types of humor production and to aimat implementing this capability in computa-tional systems. Some prototypes can produceexpressions limited in humor typology butmeant to work in unrestricted domains.11,12

Besides humor production, work has begunon humor recognition, including researchreported in this issue.13

There’s also some work on automaticemotional storytelling, where the artificialcharacter evokes empathy in the audience.14

Early results are available for automaticvideo production, especially in the simpli-fied context that doesn’t involve shooting inthe real world but starts instead from 2Dexisting images.15 Robot-based systems thatproduce good-quality pictures of events con-stitute another promising avenue.

All these themes open substantial chal-lenges for the future. They require moreinsight into emotion, cognition, and compu-tation. For example, what’s the nature of asurprise in communication and how can wemaintain attention?

SEPTEMBER/OCTOBER 2006 www.computer.org/intelligent 17

When artificial agents become

part of our environment, we

expect communication to

have the same basic properties

as human communication.

W ith so many entertainment domainsand applications, one challenge to

applying multiple intelligences to enhancethem will be creating reusable platforms thatsupport multiple entertainment activities andinteractive requirements. Figure 6 presents adigital-entertainment architecture that sup-ports models of the user, discourse, context,domain, media, and application as well as theunderlying open agent architectures andmachine learning. Models of emotions suchas fear, joy, and surprise will become impor-tant both for virtual characters and for under-standing human-user input, both satisfactionsand frustrations. Figure 6 only begins to sug-gest the complexity and opportunity richnessof interactive intelligent entertainment.

References1 A. Merlino and M. Maybury, “An Empirical

Study of the Optimal Presentation of Multi-media Summaries of Broadcast News,” Auto-mated Text Summarization, I. Mani and M.Maybury, eds., MIT Press, 1999, pp. 391–401.

2. M. Ernandes, G. Angelini, and M. Gori,“WebCrow: A WEB-Based System forCROssWord Solving,” Proc. 20th Nat’l Conf.Artificial Intelligence (AAAI 05), AAAIPress, 2005, pp. 1412–1417.

3. M. Genesereth, N. Love, and B. Pell, “Gen-eral Game Playing: Overview of the AAAICompetition,” AI Magazine, vol. 26, no. 2,2005, pp. 62–72.

4. M.T. Maybury, ed., Intelligent MultimediaInformation Retrieval, AAAI/MIT Press,1997.

5. M.T. Maybury, “Broadcast News Under-standing and Navigation,” Proc. InnovativeApplications of Artificial Intelligence, AAAIPress, 2003, pp. 117–122.

6. M. Light and M. Maybury, “PersonalizedMultimedia Information Access: Ask Ques-tions, Get Personalized Answers,” Comm.ACM, vol. 45, no. 5, 2002, pp. 54–59.

7. M.T. Maybury, ed., New Directions in Ques-tion Answering. AAAI/MIT Press, 2004.

8. W.L. Johnson, J.W. Rickel, and J.C. Lester,“Animated Pedagogical Agents: Face-to-FaceInteraction in Interactive Learning Environ-ments,” Int’l J. Artificial Intelligence in Edu-cation, vol. 11, 2000, pp. 47–78.

9. M. Cavazza and A. Simo, “A Virtual PatientBased on Qualitative Simulation,” Proc. 8thInt’l Conf. Intelligent User Interfaces (IUI03), ACM Press, 2003, pp. 19–25.

10. M. Maybury, R. D’Amore, and D. House,“Expert Finding for Collaborative VirtualEnvironments,” Comm. ACM, vol. 14, no. 12,2001, pp. 55–56.

11. K. Binsted and G. Ritchie, “ComputationalRules for Punning Riddles,” Humor, vol. 10,no. 1, 1997, pp. 25–75.

12. O. Stock and C. Strapparava, “Getting Seri-ous with the Development of ComputationalHumor,” Proc 18th Int’l Joint Conf. ArtificialIntelligence (IJCAI 03),AAAI Press, 2003, pp.59–64.

13. R. Mihalcea and C. Strapparava, “Technolo-gies That Make You Smile: Adding Humor toText-Based Applications,” IEEE IntelligentSystems, vol. 21, no. 5, 2006, pp. 33–39.

14. M.Y. Lim, R. Aylett, and C.M. Jones, “Affec-

tive Guide with Attitude,” Proc. 1st Int’l Conf.Affective Computing and Intelligent Interac-tion (ACII 05), LNCS 3784, Springer, 2005,pp. 772–779.

15. C. Callaway et al., “Automatic Cinematogra-phy and Multilingual NLG for GeneratingVideo Documentaries,” Artificial Intelligence,vol. 165. no. 1, 2005, pp. 57–89.

For more information on this or any other com-puting topic, please visit our Digital Library atwww.computer.org/publications/dlib.

I n t e r a c t i v e E n t e r t a i n m e n t

18 www.computer.org/intelligent IEEE INTELLIGENT SYSTEMS

Intelligent interaction

Tutors andmentors

User support roles

Entertainmentdomains Travel

Arts

Sports

Games and toys

Guides

Multimodal Tailored Cooperative

Expert critics

Interactive Mixed initiative

Performanceenhancers

Education

Cinema, TV, radio

Knowledge representation and inference, states, and histories

Agent architectures and machine learning

AI

Applicationmodels

Mediamodels

Taskmodel

Domainmodel

Context model

Discoursemodel

User/agentmodels

ManipulationCommunicationCognitionPerception

Figure 6. A digital-entertainment architecture.T h e A u t h o r s

Mark Maybury isexecutive director ofMITRE’s InformationTechnology Division.His research interestsinclude intelligent infor-mation systems, ana-lytic tools, collaborativeenvironments, distrib-

uted computing, and knowledge management. Hereceived his PhD in artificial intelligence fromCambridge University. Contact him at Informa-tion Technology Div., MITRE, 202 Burlington Rd.,Bedford, MA 01730; maybury@mitre.org.

Oliviero Stock is asenior fellow at theTrentino Cultural Insti-tute’s Center for Scien-tific and TechnologicalResearch (ITC-IRST).His research interestsinclude AI, natural lan-guage processing, intel-

ligent user interfaces, and cognitive technologies.He received his education at the University of Flo-rence and the University of Pisa. Contact him atITC-IRST, Via Sommarive 18, 38050 Povo,Trento, Italy; stock@itc.it.

Wolfgang Wahlsteris the director and CEOof the German ResearchCenter for ArtificialIntelligence (DKFI) anda professor of computerscience at Saarland Uni-versity in Saarbrücken,Germany. His research

interests center on language technology andintelligent user interfaces. He received his doc-toral degree in computer science at the Univer-sity of Hamburg. Contact him at DFKI GmbH,Stuhlsatzenhausweg 3, 66123 Saarbrücken, Ger-many; wolfgang.wahlster@dfki.de.