Upload
nirmala-last
View
1.517
Download
2
Tags:
Embed Size (px)
Citation preview
© W. Wahlster, DFKI
IUI99, International Conference on Intelligent User InterfacesLos Angeles, January 6th, 1999
Agent-based Multimedia Interaction for Virtual Web Pages
German Research Center for Artificial Intelligence, DFKI GmbH
Stuhlsatzenhausweg 366123 Saarbruecken, Germany
phone: (+49 681) 302-5252/4162fax: (+49 681) 302-5341e-mail: [email protected]
WWW:http://www.dfki.de/~wahlster
Wolfgang Wahlster
© W. Wahlster, DFKI
What are Virtual Web Pages?
What are Agent-Based User Interfaces?
Using Life-like Characters for the Personalization of User Interfaces
Plan-based Media Transformation and Coordination
The Combination of Retrie ved and Generated Media Objects
for the Generation of Virtual Web Pages
Commercial Applications in Advanced WebCommerce
SmartKom: a Transportable and Transmutable Multimodal Interface
Our Research Agenda for Agent-based User Interfaces
Conclusion
Outline
© W. Wahlster, DFKI
First Generation Second Generation Third Generation
Static Web Sites
Fossils cast in HTML
Interactive Web Sites
JavaScripts and Applets
Database Access andTemplate-based Generation
Dynamic Web Sites
Virtual Web Sites
Netbots,Information Extraction,Presentation Planners
Adaptive Web Sites
User Modeling,Machine Learning,
Online Layout
Three Generations of Web Sites
© W. Wahlster, DFKI
A Virtual Web Page
is generated on the fly as a combination of various media objects frommultiple web sites or as a transformation of a real web page.
looks like a real web page, but is not persistently stored.
integrates generated and retrieved material in a coordinated way.
can be tailored to a particular user profile and adapted to a particularinteraction context.
has an underlying representation of the presentation context so that an Interface Agent can comment, point to and explain its components.
Virtual Memory, Virtual Relation, Virtual Reality...
What is a Virtual Web Page?
© W. Wahlster, DFKI
Virtual Webpage Retrieved from 5 Different Servers
© W. Wahlster, DFKI
Virtual Webpage Augmented by Persona
© W. Wahlster, DFKI
PROACTIVE
ACTI
VER
EACTIVE
anticipate the user's needs adopt the user's goals provide unsolicited comments
appear as life-likecharacters
plan interactive behaviorautonomously
can initiate inter-action
INTERFACEAGENTS
respond immediatelyto interruptions
criticism and clarification questions
direct manipulation
What are Agent-based User Interfaces?
© W. Wahlster, DFKI
Consumer Provider
sells InformationGoodsServices
buys InformationGoodsServices
Web Sites
Knowledge about: Usage PatternsUser ModelsConsumer Profiles
Netbot
Intelligent Parallel Retrieval Information Extraction and
SummarizationPersonalized Presentation Matchmaking Teleshopping Assistance Telemarketing Assistance Translation Services Data Mining Services
Intelligent Web Services
© W. Wahlster, DFKI
Netbots
Indices,Directories,
Search Engines
WWW
Netbot := Intelligent Agent that uses Internet tools on a person’s behalf
Traveller’s Netbot: Tries to achieve traveller’s goals(finding and executing plans)checks availability finds best price uses personal preferences (e.g. frequent flyer programme, seating preferences lets the traveller know, when seats become available (active help)
MassServices
PersonalAssistants
e.g. Ahoy!,Jango,AiA
Netbots as Personal Assistants for WWW Users
© W. Wahlster, DFKI
Portrait Photo
Netbot: Personal Picture Finder
Parallel Meta-Search of Webpages for<Name>
<Name>
Parallel Searchin Picture ArchivesHome Pages
Extraction of Images
Filtering ofLogos, Graphics,...
WWW Knowledge Sources
Applications: Journalism, Contact Preparation, Tracing Criminals
http://finder.dfki.de:7000
A Netbot for Portrait Photos: The Personal Picture Finder
© W. Wahlster, DFKI
The Personal Picture Finder
© W. Wahlster, DFKI
Information
Information
Information
Customer 1
Provider
1
2
3
Push Technologies
+ no effort for customer
- minimal adaptation
Interactive Pull
+ good adaptation
- major effort for customerCustomer
Query
Information
Netbots with Parallel Pull
Customer NetbotQuery
Information
Provider 1
Provider 2
Provider N
+ good adaptation
+ minimal effort for customer
Customer 2
Customer N
Provider
Netbots versus Push Technologies
© W. Wahlster, DFKI
System is able to flexibly tailor presentations to the individual user and the current situation.
Enhancement of User Interfaces through Personalization
An animated characterserves as “Alter Ego”
of the presentation system.
Personalized Presenters at DFKI
© W. Wahlster, DFKI
Personalization = adaption of system behavior
according to a user model
Personalization =the “agent” appears as
animated character which presents, explains and
comments an offer, and to which the user can talk
to.
Personalized Package Presentation by an Animated Agent
© W. Wahlster, DFKI
Video Character Presents the Interior of a Boeing 757
© W. Wahlster, DFKI
Reactive Behavior of the Persona Agent
© W. Wahlster, DFKI
PersonaServer
Behaviors
Presentation GesturesReactive BehaviorsIdle-time actionsNavigation actions
Auditory Characteristics
Sound effects, auditory iconsVoice: male, female
Visual Appearances
Hand-drawnCartoonBitmaps
GeneratedBitmaps from 3D-Models
Video Bitmaps
PPP’s Persona Server implements a generic Presentation Agent that can be easily adapted to various applications
© W. Wahlster, DFKI
More than 200 cartoon frames were drawn by a professional artist.
A real persona was filmed with a video-camera and the pictures weredigitized with a frame grabber.
The frames of the visual appearance of persona can be cartoon-like images or video-frames which show real persons
© W. Wahlster, DFKI
The Persona Editor
© W. Wahlster, DFKI
take-position (t1 t2 ) point-to (t3 t4 )
move-to (t1 t2 ) r-stick-pointing (t3 t4 )
High-LevelPersona Actions
Context-SensitiveExpansion(including Navigation Actions)
Decomposition into Uninterruptable Basic Postures
r-turn (t1 t21 )
r-step (t21 t22 )
f-turn (t22 t2 )
r-hand-lift (t3 t31)
r-stick-expose (t31 t4 )
Bitmaps
... ... ... ...
Context-Sensitive Decomposition of Persona Actions
© W. Wahlster, DFKI
MultimediaGeneration
TextGenerator
GraphicsGenerator
GestureGenerator
AnimationGenerator
PresentationPlanner
(PREPLAN)
KR & Reasoning
(RAT)
ProductionActs
GeneratedMaterial
PresentationActs
Signals +Events
MultimediaServer
LayoutManager
PersonaServer
MusicGenerator
PPP System Architecture
© W. Wahlster, DFKI
Plan multimedia material as well as presentation acts and their temporal coordination
Presentation Acts
Persona Acts Display Acts
This is thetransformer
Task of the Presentation Planner
© W. Wahlster, DFKI
Persona explains a modem
© W. Wahlster, DFKI
Production ActPresentation Act
Introduce
Create-Graphics
S-Show
S-Wait
S-Position Elaborate-Parts
S-Create-Window
S-Depict
Label Label
S-Point S-Speak S-SpeakS-Point
Qualitative constraints: Create-Graphics meets S-Show, ...
Metric constraints: 1 <= Duration S-Wait <= 1, ...
Distinction between production and presentation acts
(i.e. Persona- or display acts)
Explicit representation of qualitative and quantitative constraints
Extensions of the Representation Formalism
© W. Wahlster, DFKI
(define-plan-operator:HEADER (A0 (INTRODUCE P A ?object ?window)):INFERIORS ((A1 (CREATE-GRAPHICS P A ?window ?object))
(A2 (S-SHOW P A ?window ?object))(A3 (S-POSITION P A ?window))(A4 (S-WAIT P A))(A5 (ELABORATE-PARTS P A ?object ?window)))
:QUALITATIVE((A1 (m) A2) (A3 (s) A2) (A3 (m) A5) (A5 (m) A4) (A4 (f) A2)):METRIC((10 <= DUR A2) (2 <= DUR A4 <= 2)):START A1:FINISH A2)
(cf. [André/Rist 97])
Presentation Strategies in PPP contain qualitative and metric constraints
© W. Wahlster, DFKI
PRELIMINARY SCHEDULE
PPP first builds up a preliminary schedule at design time
© W. Wahlster, DFKI
UPDATED SCHEDULE
The preliminary schedule is continously updated at presentation time
© W. Wahlster, DFKI
Temporal
Consistency
Checker and
Propagator
Plan
Scheduler
Plan Nodes with
Links to Local
Temporal
Constraint
Networks
Plan Operators
with Metric and Qualitative
Temporal Constraints
Temporal Reasoning in the Presentation Planner for Dynamic Multimedia Coordination
© W. Wahlster, DFKI
Persona Presents an Automatically DesignedBusiness Chart
© W. Wahlster, DFKI
Multi-DomainProblem Specs
NETBOT
RetrievedResults
Information Structures Relations, Lists KR Terms
Media Objects Texts, Sounds, Videos Pictures, Maps,
Animations
DistributedInformation
MultipleData Sources
The Combination of Retrieved and Generated Media Objects for Virtual Webpages
© W. Wahlster, DFKI
RetrievedResults
Select & Design
Select CannedMedia Objects
Design NewMedia Objects
Information Structures Relations, Lists KR Terms
Graphics,Animation
Text, Speech,Mimic
Icons, Clip Art
Frames, Sounds
Reuse & Transform
CoordinateMedia Objects
TransformMedia Objects
Temporal Synchroni-zation
SpatialLayout
Clip, Convert,Abstract
Zoom, Pan,TransitionEffects
Media Objects Texts, Sounds, Videos Pictures, Maps,
Animations
The Combination of Retrieved and Generated Media Objects for Virtual Webpages
© W. Wahlster, DFKI
Virtual Webpage with Animation Effects Based on a Single GIF Image
© W. Wahlster, DFKI
Transition Effects in a Series of Retrieved Pictures
© W. Wahlster, DFKI
Persona as a Personal Travel Consultant
© W. Wahlster, DFKI
NetbotPAN
Trip Data
Pictures andGraphics
Pieces of Text
Coordinates forPointing Gestures
Input for SpeechSynthesis
Icons forHyperlinks
HotelAgent
MapAgent
Address
WeatherAgent
Train & FlightScheduling
Agent
Major EventAgent
VirtualWeb
Presentation
Constraint-basedOnlineLayout
PresentationPlanner
PersonaServer
Componentsof virtual
Webpages
AiA
The Generation of Virtual Webpages with PAN and AiA
© W. Wahlster, DFKI
Persona as a Personal Travel Consultant
© W. Wahlster, DFKI
S-Include-Photo
S-Include-Text
S-Include-Link
Introduce
S-SpeakIllustrate
Design-Intro-Page Emphasize
This hotel has a nice swimming
pool.Location
Elaborate
S-Speak S-Point
S-Include-Map Label
Your hotel is located here.
Location Link Selected
Default Time Over/ Up
Dynamic Node Expansion for the Conditional Generation of Virtual Web Pages
© W. Wahlster, DFKI
Use of a Life-like Character for Electronic Commerce
© W. Wahlster, DFKI
Use of a Life-like Character for Electronic Commerce
© W. Wahlster, DFKI
Use of a Life-like Character for Electronic Commerce
© W. Wahlster, DFKI
Jennifer James as a Virtual Sales Agent
© extempo systems inc.
© W. Wahlster, DFKI
Plug-Ins Applets
Add features (character players) to browser Download triggered by user Requires disk space on client Unrestricted access to client Less appropriate for WebCommerce,
Guides Agents integrated in 3D environments Appropriate for Entertainment
Examples:
Extempo's Jennifer James (Hayes-Roth et al. 98)
PFMagic's virtual petz
Java animation code sent over the net Automatic loading Requires no disk space on client Restricted access to client Appropriate for WebCommerce, Guides Agents integrated in 2D environments Less appropriate for Entertainment
Examples:
DFKI's Web Persona(Müller et al. 98)
ISI's Adele (Johnson et al 98)
New in AiA/PAN: Balanced combination of Applets and ServeletsEfficient distribution of client-side Java and server-side Java for driving theInterface Agent
Sending Interface Agents to Clients: Plug-Ins or Applets?
© W. Wahlster, DFKI
Queries
TransactionsShopbots
2Provider
pays usage fee 3 pays fee after successfulcommercial transaction
InternetShop 1
.
.
.
.
.InternetShop n
Provider1
pays fee forbanner avertisement
1
2
3
Banner fee
Usage fee
Transaction fee (credit card model)
Alternative Business Models for Shopbots
© W. Wahlster, DFKI
Advanced WebCommerce
Advanced WebCommerce
Virtual WebPages
One-to-OneMarketing
One-to-OneMarketing
Intuitive,Multilingual
Access
Dialogue withVirtual Sales
Agents
Shopbots forAutomated
ComparisonShopping
Text Analysis and Generation
User Modeling and Language GenerationCoordinated Text & Graphics Planning
Robust Dialogue Understanding
Advanced SpeechSynthesis
Information Extraction from HTML/XML Documents
MachineTranslation
MultimodalInterfaces
MultimediaPresentation
Planning
Intelligent Interface Technology is a Prerequisitefor Advanced WebCommerce
© W. Wahlster, DFKI
aa
Mensch-Technik-Interaktion
HOME/OFFICE
aaa
Mensch-Technik-Interaktion
aa
Mensch-Technik-Interaktion
SmartKom-Home/Office:A Versatile Agent-based Interface
SmartKom-Public: A Multimodal
CommunicationBooth
SmartKom-Mobile: A Handheld
CommunicationAssistant
Media Analysis
Kernel ofSmartKomInterface
Agent
Interaction Management
ApplicationManage-
ment
MediaDesign
SmartKom: A Transportable and Transmutable Interface Agent
© W. Wahlster, DFKI
MediaInterfaceEuropean Media LabUinv. Of
MunichUniv. ofStuttgart
Saarbrücken
Aachen
Dresden Berkeley
Stuttgart
MunichUniv. ofErlangen
Heidelberg
Main ContractorProject Management
TestbedSoftware Integration
DFKISaarbrücken
The SmartKom Consortium:
Project Budget: $ 34 MProject Duration: 4 years
DAIMLERCHRYSLERUlm
SmartKom: Intuitive Multimodal Interaction
© W. Wahlster, DFKI
Smartcard/ Credit Cardfor authentication and billing
Docking stationfor PDA/Notebook/Camcorderhigh speed and broadbandwidth Internet connectivity
High-resolution scanner
Loudspeaker
Room microphone
Face-tracking camera
Virtual touchscreenprotected against vandalism
Multipoint video conferencing
SmartKom-Public:A Multimodal Communication Booth
© W. Wahlster, DFKI
MOBILE
Camera
GPS
Microphone
Loudspeaker
Stylus-Activated Sketch Pad
WearableComputeServer
Docking Stationfor Car PC
Biosensorfor Authentication
& Emotional Feedback
GSM for Telephone,Fax, Internet Connectivity
SmartKom-Mobile: A Handheld Communication Assistant
© W. Wahlster, DFKI
SpeechMikeVirtual Touchscreen
Natural Gesture Recognition
SmartKom-Home/Office: A Versatile Agent-based Interface
© W. Wahlster, DFKI
User(s)
MediaAnalysis
Design
Media Fusion
OutputRendering
Representation and Inference
UserModel
DiscourseModel
DomainModel
TaskModel
MediaModels
InteractionManagement
MediaAnalysis
InputProcessing
Information
Applications
People Intention
RecognitionMediaDesign
Ap
pli
cati
on
In
terf
aceDiscourse
Modeling
UserModeling
PresentationDesign
Language
Graphics
Gesture
Biometrics
Language
Graphics
Gesture
AnimatedPresentation
Agent
The Architecture of the SmartKom Agent (cf. Maybury/Wahlster 1998)
© W. Wahlster, DFKI
Personalized Presentation Agents (limited user interaction)
WIP: 1989-1993
1
Personalized Interface Agents (full user interaction)
2PPP:1994-1996
Multiple Interface Agents (agent-agent and user interaction)3
Multiple Presentation Agents in one scene (eg. pros and cons)
Multiple Role-Taking (eg. Travel Assistance vs. Comparison Shopper)
Multiple Interface Agents (eg. human-computer, human-human interaction)
AiA: 1997-2000
Our Research Agenda for Agent-based Interfaces(Wahlster, André, Rist, Müller, Graf etc; www.dfki.de/imedia)
© W. Wahlster, DFKI
I recommend you this SLX limousine.
Multiple Agents Discussing Pros and Cons of a Mercedes Model
© W. Wahlster, DFKI
IntelligentWeb
Services
Intelligent Web
Services
UserModeling
Planning
Natural LanguageUnderstanding
KnowledgeRepresentation
ImageUnderstanding
MachineLearning
PlanRecognition
InformationRetrievalMultimodal
UserInterfaces
Research on Intelligent Web Services brings disparate subfields in the area of intelligent systems together
© W. Wahlster, DFKI
ECommerce projects of DFKI have shown that research on agent-based multimodal interfaces can be transferred to real real-world applications:
Dekra (largest European organization of used car dealers): FairCar as an ECommerce platform with NL access and a comparison shopping agent for used cars
DaimlerChrysler: IKP for online user modelling in a one-to-one marketing system for Mercedes cars
Otto/Shopping24/Eddie Bauer (largest European mail order company): virtual sales agents for one-to-one marketing of fashion and computer hardware
Lufthansa/Condor: direct marketing of charter flights
Conclusion
© W. Wahlster, DFKI
Two Research Challenges:
Making the interface agents sensitive to temporary limitations of the user´s time and working memory capacity (cf. our READY project, Jameson et al., p. 79-85 in IUI99 Proceedings)
Making the agents instructible, so that they can learn from the user in a dialog, how to extract information in a new domain (cf. our PAN project, Bauer/Dengler, p. 153-156 in IUI99 Proceedings)
The generation of virtual web pages by agent-based multimodal interfaces leads to innovative applications in:
Electronic Commerce, Electronic TV Guides (EPG)Telelearning environments, Call Centers and Help Desks
Conclusion