Arne Jönsson 1

Multimodal research in Linköping

Arne Jönsson

NLPLAB/HCS/IDA

Arne Jönsson 2

NLPLAB

Upper row:Pernilla,Sonia,Lena,Annika,Micke,Magnus

Bottom row:Håkan,Lars A,Robert,Mustapha,Arne,Lise,Anders,Lars D

Missing:Nils, Bertil, Pontus, Gen

Arne Jönsson 3

Current research

• (Multimodal) information access- Natural language interaction- Text and document analysis

• Usability of multimodal interfaces

• Open-Source and development methods

Arne Jönsson 4

Dialogue system

Dialoguemanager

Generator

Interpreter

StructuredInformation

Dialogue system

Arne Jönsson 5

Ex TV programme guide

• Search for TV programs

• Mediaterminal

• Structured database

• Spoken input

• Developed from framework and method for iterative development

• With Nokia Home Communication

Arne Jönsson 6

Demo

Arne Jönsson 11

Text and document analysis

Language analysis

StructuredInformation

Unstructured documents

Q&A

MT

Arne Jönsson 12

Ex. AskRSV

• Information extraction from text document

• Simple Q&A-system

• With Ida Infront

Arne Jönsson 13

Ex. taxation brochure

24 pages unstructured text

Arne Jönsson 14

Domain ontology in ASK-RSV

• Taxonomy of a limited set of objects, events, properties and relations in the tax domain

• Only IS-A and synonymy relations

OBJECT

THING

ACTOREVENT

PAPER DATE TAXATION TAX EVENT ORG.PEOPLE

TAX-RETURN ASSET DEBT INCOME TRANSACTIONCOMP. PUBL

Arne Jönsson 18

Combined system

StructuredInformation

Unstructured documents

Language analysis

Dialoguesystem

Lexicon Grammar Ontology

Arne Jönsson 19

Shared knowledge sources

• Ontology, lexicon and grammar- Different demands from IE and DS

• Design

• Development- Corpus analysis, user - Background system, database

• Automatic update

Arne Jönsson 20

BirdQuest

• Bird encyclopaedia- Tagged, templates

• Corpus with user questions- Analysed

• Based on dialogue systems framework

Arne Jönsson 26

Ötraf

• Time tables from local bus company

• Multimodal system- Dialogue- Handling of spatial and temporal

expressions

Arne Jönsson 28

Studies of multimodal interaction

• Wizard of Oz-simulation • Feedback study

- User’s experience- Interviews, questionnaires- Various dimensions: control, cooperation, ease of use,

efficiency, anxiety, hability etc

- Three conditions- No spoken feedback- Limited spoken feedback- Full spoken feedback

- 30 subjects

Arne Jönsson 29

Results

• All or nothing• More like a human in the full feedback

condition- Before the study subjects wanted to have a

timetable for the task, after they preferred a human

• No difference in experienced control and cooperation

• The speech synthesis better in full feedback mode- Same voice

Arne Jönsson 30

Open source

• Share research results between research groups- Stimulate exchange and co-operation

• Language technology readily available for industry- Increased use and insights- The companies do their own prototype

development• Robust code• nlpFarm

http://sourceforge.net/projects/nlpfarm/

Arne Jönsson 31

http://sourceforge.net/projects/nlpfarm/