02/04/09Danica Damljanović1 Natural Language Interfaces to conceptual models: usability and performance Danica Damljanović [email protected]

02/04/09 Danica Damljanović 1

Natural Language Interfaces to conceptual models: usability and performance

Danica Damljanović

[email protected]

University of Sheffield NLP

Outline

• NLIs to KBs and their usability

• QuestIO – Question-based Interface to Ontologies Demo and evaluation

• Towards better usability using FREYA

• Conclusion

202/04/09 Danica Damljanović


Natural Language Interfaces

• Allow users to interact with a system using written or spoken language to perform tasks which require knowledge of a formal query language NLIs to structured data:

• NLIs to DBs: TEAM, PRECISE• NLIs to KBs

NLIs to semi-structured data• Open-domain question answering systems

Other:• E.g. NLC: replacement for a programming language• Dialog and tutoring systems


Danica Damljanović

NLIs to knowledge bases

(Kaufmann and Bernstein, 2007)

• Natural Language Interfaces preferred to keywords, menu-guided, and graphical interfaces

(Linckels, 2007): • keywords preferred

to NL interfaces


8Danica Damljanović

NLIs to KBs



Usability of NLIs

• Who uses NLIs? Application developers: customisation End users: search

• Usability Effectiveness Efficiency User satisfaction



Usable NLIs to KBs: challenges

• Robustness

• Portability

• What to show?

• Understanding information need

• Habitability



Habitability

• Can the User and the System speakthe same language?



Question-based Interface to Ontologies



NL --> SPARQL query

Filtering concepts

Ranking concepts

Query Creator

Query Execution


An Example

1.15

1.19

compare


Demo

• http://www.tao-project.eu/researchanddevelopment/demosanddownloads/movies/questio/questio.html

• http://www.tao-project.eu/researchanddevelopment/demosanddownloads/movies/prototype-tutorial/prototype-tutorial.html



User-centric task-based evaluation

• Training: using video tutorials

• 12 participants, 4 tasks: 3 defined, e.g. “Find runtime parameters of

Cebuano gazetteer.” 1 free task: “Think of any task that you would

like to perform using this prototype.”

• Measured: Efficiency, Effectiveness and User satisfaction



Efficiency: average time per task


Effectiveness: how successfully the tasks were finished?

University of Sheffield NLP Post –task survey: question 1


Post –task survey: question 2






Evaluation: conclusion• Browsing ontology mostly helpful – but

only for users who are familiar with ontologies

• Refinement pane: primitive, still favourable for defined tasks

• Tasks not finished: in cases when the user information need was

not precisely expressed (undefined tasks) When the answer was not in the ontology


University of Sheffield NLP “find out which are the runtime parameters of Cebuano Gazetteer”

• “cebuano gazetter parameters”• “ What are the runtime parameters of cebuano

gazetteer?“• “what are the parameters of cebuano gazetteer?”• “Cebuano gazetteer runtime parameters“• “Runtime parameters of cebuano gazetteer“• “Cebuano runtime parameters“• “Cebuano gazeteer“>>“Cebuano gazetteer“

Find parameters by browsing the ontology35Danica Damljanović


“runtime parameters of Cebuano Gazetteer”

• “Cebuano runtime parameters“GATE plugin Resource Parameter

Processing Resource

Contains

Resource hasInitTimeParameter

hasRuntimeParameter


Undefined tasks

• “Developer of Tokeniser”• ”Projects about GATE ”, ”GATE web site”• ” Tokenizer ” vs. ” Tokeniser ”• ” Gazetteer ” vs. ”Gazeteer ”• ”horacio saggion publications” vs. ”horacio

saggion articles”• ”Author of morphological analyser ” >>

”Developer of morphological analyser ” >> ”Developer and morphological analyser ”



Next steps: FREyA


FREyA (Feedback, Refinement, Extended vocabulary, Agregation)


Example

• the user's query: Cities in Europe

• FREyA: Cities >>locatedIn >>Countries >> locatedIn >>

Europe Cities >>locatedIn >> Countries >> partOf >> Europe

• Attempto: which cities are located in countries that are

part of Europe?



Challenges

• Return answers in real time: which rivers flow through Germany >>

which rivers flow through (cities in) Germany

• Refinement model - an ontology? with concepts such as query (hasAnswer)

answer, refinement, feedback, subject of query (main subject), domain, context.

find out what the user wants and instantiate the refinement ontology.




Expected contribution

• Usability improvement of NLIs: Eliminate training Improve the performance by implementing

user-system interaction using FREYA:• help the user to easily familiarize himself

with the system capabilities• express his need more precisely, in a way

which is understandable by the system



Thank you

• Questions?


References• (Noy et al., 2001)N. Noy, M. Sintek, S. Decker, M. Crubezy, R. Fergerson, and M.

Musen. Creating Semantic Web Contents with Protege-2000. IEEE Intelligent Systems, 16(2):60-71, 2001.

• (Kiryakov et al., 2004) A. Kiryakov, B. Popov, D. Ognyano, D. Manov, A. Kirilov, and M. Goranov. Semantic annotation, indexing and retrieval. Journal of Web Semantics, ISWC 2003 Special Issue, 1(2):671-680, 2004.

• (Guha et al., 2003) R. Guha, R. McCool, and E. Miller. Semantic search. In WWW '03: Proceedings of the 12th international conference on World Wide Web, pages 700-709, New York, NY, USA, 2003. ACM.

• (Lei et al., 2006) Y. Lei, V. Uren, and E. Motta. Semsearch: a search engine for the semantic web. In Managing Knowledge in a World of Networks, pages 238{245. Springer Berlin /Heidelberg, 2006.

• (Kaufmann & Bernstein, 2007) E. Kaufmann and A. Bernstein. How useful are natural language interfaces to the semantic web for casual end-users? In Proceedings of the Forth European Semantic Web Conference (ESWC 2007), Innsbruck, Austria, June 2007.

• (Serge Linckels, 2007) C. M. Serge Linckels. Semantic interpretation of natural language user input to improve search in multimedia knowledge base. it - Information Technologies, 49(1):40-48, 2007.

Documents

02/04/09Danica Damljanović1 Natural Language Interfaces to conceptual models: usability and performance Danica Damljanović [email protected]