Upload
wilsmith73
View
284
Download
1
Embed Size (px)
DESCRIPTION
Constructing an AI knowledge base requires decomposing complex sentences into simplified statements with encoding concepts. Due to knowledge engineering cost and complexity, we created an experiment to test the scenario where college students do the above task using a semantic wiki. This wiki also tracked the progress of each student and provided an integrated environment for our knowledge workers. In this presentation we will discuss the layout of the imported data within the wiki, the user experience throughout the publishing process, the underlying technologies behind the wiki app, and the preliminary results of the experiment. The semantic wiki web application included the following technologies: • Semantic MediaWiki Plus, which provides an object oriented framework for semi-structured data. • JavaScript, HTML5, and AJAX service-based graphing of triples and entities within the project and for interconnected services. • Faceted browsing and semantic pivoting among related entities: textbook paragraphs, sentences, concepts, and sentence encodings. • Virtuoso integration with the knowledge base.
Citation preview
AUTHORING WITH AURA WIKISemTechBiz 2013, San Francisco
… A CASE STUDY
Today we will be talking about…
• Populating a Symbolic AI – Aura• The spiraling cost structure for encoding data into a symbolic AI
• How do we bring low cost domain experts into the process?
• Creating a Semantic MediaWiki Installation• Importing a textbook into Semantic MediaWiki and marking up pages with properties
• Customizing the installation for annotating textbook sentences
AURA
3) Encoding Planning -- 35% time
Group Common UTs, ID KR/KE Issues,
ID Already Encoded, Write How to Encode
Pre-Planning, QA Check
Status Labeling: Encoding Complete, KR Issue (Closed)
2) Reaching Consensus -- 14% timeUniversal Truth Authoring, Concept Chosen QA Check
1) Determining Relevance -- 2% timeHighlighting, Diagram Analysis
QA Check
Status Labeling: Relevant, Irrelevant (Closed)
6) Question-Based Testing -- 14% timeUse Minimal Test Suite, Reasoning JIRA Issues Filed,
Encoder Fills KB GapsQA Check with Screenshots of “Passing" Comparison
and Relationship Questions
5) Key Term Review -- 25% timeKR Evaluated by Modeling Expert and Biologist,
Encoder Makes ChangesKR Evaluated by Modeling Expert and Biologist
QA Check
4) Encoding -- 10% timeEncode, File JIRA Issues
QA Check
Status Labeling: Encoding Complete, KE Issue
-- How to choose a concept given a UT?-- How to produce UTs from sentences?
Sentence
Sentence
UT
UT
UT
UT
Chapter
Chapter
KBBook
CMap
CMap
CMap
CMap
Chapter UT
2) Reaching Consensus -- 14% time
Univeral Truth Authoring, Concept Chosen
What is a Universal Truth?• “A Universal Truth is a stand-alone, unambiguous
declarative sentence about a textbook topic that expresses a single fact that is universally true”- AURA Knowledge Engineering Manual
• “Water is composed of two Hydrogen element molecules and one Oxygen element molecule with the chemical formula H20”• Water is composed of hydrogen• Water is composed of oxygen• Hydrogen is an element• Oxygen is an element• Water has the chemical formula H20
• Does: “Water is a compound” count?
Project Goals
• “Crowd Source Universal Truth Authoring”• Can Domain Experts Author Useful Universal
Truths?• Can We Speed Up Encoding a Textbook with Input
from Domain Experts?• Can We Create a UT Authoring Portal for Multiple
Textbooks?• Can Existing Social Networks Provide Domain
Experts Capable of UT Authoring?• Could Gamification be Applied to An Existing Portal
to Add Non-Domain Experts?
About the Domain Experts
• Students attending University of Washington or recent graduates
• All have a background in biology or life sciences• Native English speakers with excellent writing skills
• Each student read the chapters in question and was provided with an iPad running the Inquire application
• Students were paid for their time
A Semantic MediaWiki Portal
Storing a Text Book in Aura Wiki• The wiki was created with instances of page types
composed of textbook sentences• Sentence• Paragraph• Section• Chapter• Book
• The wiki also has imported resources to aid in the UT authoring process• Glossary Pages• Taxonomy Concepts• Universal Truths – Human and Machine
Navigating Aura Wiki
The text isn’t
readable
Is this the table of contents?
Why didn’t
I see this
first?Where’s the next
sentence?THIS PAGE HAS
NO FORMATTING
AND LOOKS
NOTHING LIKE
THE TEXTBOOK!
Navigating Aura Wiki
Removed
to Google
Analytics
Authoring Universal Truths
• Components : • Read Sentence• Access Sentence Context• Access Neighboring
Sentences• Check & Submit
Relevancy• Check & Submit Authoring
Status• Display Existing Universal
Truths• Author Universal Truths
Authoring Universal Truths
• Semantic Wiki Properties• Each page has a unique id
for the table of contents element
• The sentence itself is an element
• Elements pointing to the previous and next sentences.
• Elements pointing to top level entities
• Users can update the sentences relevancy and encoding status.
Sentence and Context View
Authoring Universal Truths
Input form for new UT. First two inputs are
required.
Authoring Universal Truths
• Semantic Wiki Properties• Reference sentence• The universal truth text• UT concept – AURA provided• UT context – AURA provided• Accuracy rating for the universal
truth• Date created, approved, and
when ratings were applied
Universal Truth
PROPOSALSUser Experience Review
… 5 of 5,000
Navigating Aura Wiki• Unregistered and Registered Main Pages
• Unregistered users are locked out• Registration is turned off for anonymous users
• Unique Extensions Proposed for Guided Authoring
How to View a Textbook Paragraph?Get rid of
title id
elements
Context needs to
include more
Click on
sentences in
context to
annotate next
Auto create triple format UTs from
sentence?
How to View a Universal Truth Page?
Can you edit
a UT you
didn’t create?
Is this edit another version
of the wiki page?
Is rating a UT
another
version of the
page?
How do we unify versions of the page for export
to AURA?
Knowledge Engineer Editing
Faceted
Browsing by
properties
Bulk editing by sentence /
propertyBulk
moderation of
universal
truths
Knowledge Engineer Editing
STUDENT REVIEWCan Experts Author Universal Truths?
Domain Expert Authoring Statistics• 6 University of Washington Students participated in the
test• Each received 45 minutes of training on creating
Universal Truths• Each was given 1 hour and a pre-selected list of
sentences on a user page to complete• The groups generated over 100+ Universal Truths each
session• They averaged 37 Universal Truths an hour per student• Students were frequently observed using their domain
experience to construct UTs not specifically worded in the source sentence (ie: “Water is a compound”)
CONCLUSION
Project Goals
• “Crowd Source Universal Truth Authoring”• Can Domain Experts Author Useful Universal Truths?
• Can We Speed Up Encoding a Textbook with Input from Domain Experts?
Project Goals
• “Crowd Source Universal Truth Authoring”• Can We Create a UT Authoring Portal for Multiple Textbooks?
Project Goals
• “Crowd Source Universal Truth Authoring”• Can Existing Social Networks Provide Domain Experts Capable of UT Authoring?
• Could Gamification be Applied to An Existing Portal to Add Non-Domain Experts?
QUESTIONS? COMMENTS?
THANK YOU(clap now)