Upload
others
View
3
Download
0
Embed Size (px)
Citation preview
NL-‐Graphs: A Hybrid Approach toward Interac9vely Querying Seman9c Data
Khadija Elbedweihy Suvodeep Mazumdar Stuart N. Wrigley Fabio Ciravegna
OAK Group University of Sheffield
Sheffield, UK
k.Elbedweihy | s.mazumdar | s.wrigley | f.ciravegna | @ shef.ac.uk ovus00
Users and User Experience
• Seman9c Search System • Assist users sa9sfy their informa9on needs
• User Sa9sfac9on – Query approach – Performance of Search system – Presenta9on of results – ….
• Usability and User Experience largely ignored – Need to focus on users
Query Approaches
• Types • Interac9on mechanisms • Formalisms (Natural language – Forms – Formal Query)
• O[en Task or applica9on Specific – Keyword-‐based approach to quickly iden9fy possible relevant
instances – Formal query for more complex tasks (nega9on, group-‐by etc.)
• Most SW applica9ons focus on one approach
Esther Kaufmann and Abraham Bernstein. Evalua9ng the usability of natural language query languages and interfaces to seman9c web knowledge bases. Journal of Web Seman/cs, 8(4):377–393, 2010.
Spectrum of Query Approaches
4
Issues + Mo9va9ons • Comparison of Query Approaches (Expert and Casual) [2]
§ NL Approach + Prior experience with Google-‐like approaches + Easy and quick to write ques9ons/keywords - Less support for Complex queries
§ View Based Approach + Good support in complex queries + Visualising the search space - Time and effort required formula9ng queries
• Learnability study with Graph based approach (Experts) [3] + Improvement in user sa9sfac9on and performance over 9me - S9ll needed 9me and effort to formulate queries - Extensive use of keyword-‐search for concepts
[2] K. Elbedweihy, S. N. Wrigley, and F. Ciravegna. Evalua9ng Seman9c Search Query Approaches with Expert and Casual Users. In Proceedings of the Evalua9ons and Experiments Track, ISWC, 2012. [3] S.Mazumdar, D.Petrelli, K. Elbedweihy, V. Lanfranchi, and F. Ciravegna. Affec9ve Graphs: The visual appeal of linked data. Seman9c web : interoperability, usability, applicability, 2013. In Press.
Affec9ve Graphs
Affec9ve Graphs
• Demo
HybridGraphs
• NL [4] + Graph [3] = Hybrid Query Approach • Builds on Advantages of either approach
• Quick access to relevant data • Easy and visual means to formulate complex queries • Visually inspect search space
• Alleviates the disadvantages • Lesser 9me taken to build complex queries
[3] S.Mazumdar, D.Petrelli, K. Elbedweihy, V. Lanfranchi, and F. Ciravegna. Affec9ve Graphs: The visual appeal of linked data. Seman9c web : interoperability, usability, applicability, 2013. In Press. [4] K. Elbedweihy, S.N. Wrigley, F. Ciravegna, and Z. Zhang (2013). Using BabelNet in bridging the gap between natural language queries and linked data concepts. NLP & DBpedia Workshop, 12th Interna9onal Seman9c Web Conference (ISWC2013), 21-‐25 October, Sydney, Australia.
Hybrid Approach
Evalua9on • Previous evalua9ons on NL component and Affec9ve Graphs [3,4]
(performance, usability, effec9veness etc.) • Focus on hybrid approach as a new query mechanism and the
support provided • Methodology
– Expert (SW experts in Computer Science + So[ware development firm) and Casual users (staff and students at the University of Sheffield)
– 24 subjects (12 + 12) – Aged between 18 and 53, mean age 31 years – Short demonstra9on of a scenario based example – Familiarisa9on with the system
[3] S.Mazumdar, D.Petrelli, K. Elbedweihy, V. Lanfranchi, and F. Ciravegna. Affec9ve Graphs: The visual appeal of linked data. Seman9c web : interoperability, usability, applicability, 2013. In Press. [4] K. Elbedweihy, S.N. Wrigley, F. Ciravegna, and Z. Zhang (2013). Using BabelNet in bridging the gap between natural language queries and linked data concepts. NLP & DBpedia Workshop, 12th Interna9onal Seman9c Web Conference (ISWC2013), 21-‐25 October, Sydney, Australia.
Evalua9on Tasks
• 5 ques9ons derived from QALD dataset [5] • Different levels of difficulty
– 1. When was Capcom founded? – 2. What did Bruce Carver die from? – 3. Who was the wife of U.S. president Lincoln? – 4. Give me all ci9es in Alaska with more than 10000 inhabitants. – 5. Show me all songs from Bruce Springsteen released between 1980
and 1990.
[5] V. Lopez, C. Unger, P. Cimiano, and E. Moqa. Evalua9ng ques9on answering over linked data. Web Seman9cs: Science, Services and Agents on the World Wide Web, 2013.
Evalua9on Analysis
• Objec9ve and Subjec9ve data collected • Input Time, number of aqempts, Success rate • Post evalua9on ques9onnaires – System Usability Scale (SUS),
Extended ques9onnaire – The query construc9on process was X. (5-‐point Likert scale, ranging
from Laborious to Effortless) – What did you like about the hybrid approach as a mechanism for
expressing your query? and why? – What things you didn't like about the hybrid approach as a mechanism
for expressing your query? and why? • Aesthe9cs ques9onnaire
Results
• Overall, experts (73.75) liked NL Graphs more than casuals (61.25) – Graph based approach more complicated – Experts more familiar with Seman9c Web and graph data
• Success rate 100% (all users were able to successfully complete the tasks) mostly on one aqempt
• Query Construc9on process scored effortless • Most users commented posi9vely on the usability and support
provided by the hybrid approach • Four users (1e + 3c)
– Complicated and non-‐intui9ve – Why are there more steps required to build queries than with Google? – “It seemed an extra step to get to your answer rather than just typing
in a search and it appearing in results.”
Results Contd.
• More assistance needed/desired • Need for more Training
• Experts took longer (94.48s) to formulate queries than casual (76.88s) users – Follow logic and prior understanding of SW concepts – Increased interest in the interface, trying out new features, exploring
technical aspects (e.g. SPARQL query generated)
Results Contd.
Results Contd.
• Q4 – Give me all ci9es in Alaska with more than 10000 inhabitants
– Use of terms like “Alaskan” – DbPedia proper9es such as isPartOf – Numerical constraints >10000
• Q5 -‐ Show me all songs from Bruce Springsteen released between 1980 and 1990 – Sezng dates as a constraint is a manual process
Summary
• Hybrid approach to querying Linked Data • It works ! J • High SUS scores indicate usability and usefulness • Future work
– Several re-‐designs iden9fied/suggested in the evalua9on – A longer term study directly comparing Hybrid approach to other
approaches in iden9cal condi9ons – Study with larger set of user groups – Improving NL system
[3] S.Mazumdar, D.Petrelli, K. Elbedweihy, V. Lanfranchi, and F. Ciravegna. Affec9ve Graphs: The visual appeal of linked data. Seman9c web : interoperability, usability, applicability, 2013
Aesthe9c design in Seman9c Web Applica9ons [3]
Thank You
• Ques9ons ?
Hybrid Approach
• Usability + Interac9vity • NL [4] + Graph [3] = Hybrid Query Approach • Builds on Advantages of either approach
– Quick access to relevant data – Easy and visual means to formulate complex queries – Visualise nearby concepts – Visually inspect search space
• Alleviates the disadvantages – Lesser 9me taken to build complex queries
[3] S.Mazumdar, D.Petrelli, K. Elbedweihy, V. Lanfranchi, and F. Ciravegna. Affec9ve Graphs: The visual appeal of linked data. Seman9c web : interoperability, usability, applicability, 2013. In Press. [4] K. Elbedweihy, S.N. Wrigley, F. Ciravegna, and Z. Zhang (2013). Using BabelNet in bridging the gap between natural language queries and linked data concepts. NLP & DBpedia Workshop, 12th Interna9onal Seman9c Web Conference (ISWC2013), 21-‐25 October, Sydney, Australia.
Quan9ta9ve Evalua9on
• Query Execu9on phase (98ms to 19.9s)
20
Evalua9ng.views. – Performance Test Results
21
Remote database Test Results – Variability between. 0.00026 ms to 6.48ms
10 Principles of Aesthe9cs for SW
• Review of literature 10 general guidelines &
22
Mazumdar, S., Petrelli, D., Elbedweihy, K., Lanfranchi, V., & Ciravegna, F. Affec9ve Graphs: The Visual Appeal of Linked Data. Seman9c Web: Interoperability, Usability, Applicability.
9 node-‐linked visualisa9on guidelines
Measuring Aesthe9cs
• Screenshots of interfaces fed-‐in to a Java program, where each element was “marked-‐up” using mouse gestures
24
D.C.L. Ngo, A. Samsudin, and R. Abdullah. Aesthe9c measures for assessing graphic screens. J. INF. SCI. ENG, 16(1):97–116, 2000. D.C.L. Ngo, L.S. Teo, and J.G. Byrne. Modelling interface aesthe9cs. Informa9on Sciences, 152:25–46, 2003.
Results
25
0.652
0.561