31
AUTHORING WITH AURA WIKI SemTechBiz 2013, San Francisco A CASE STUDY

AURA Wiki - Knowledge Acquisition with a Semantic Wiki Application

Embed Size (px)

DESCRIPTION

Constructing an AI knowledge base requires decomposing complex sentences into simplified statements with encoding concepts. Due to knowledge engineering cost and complexity, we created an experiment to test the scenario where college students do the above task using a semantic wiki. This wiki also tracked the progress of each student and provided an integrated environment for our knowledge workers. In this presentation we will discuss the layout of the imported data within the wiki, the user experience throughout the publishing process, the underlying technologies behind the wiki app, and the preliminary results of the experiment. The semantic wiki web application included the following technologies: • Semantic MediaWiki Plus, which provides an object oriented framework for semi-structured data. • JavaScript, HTML5, and AJAX service-based graphing of triples and entities within the project and for interconnected services. • Faceted browsing and semantic pivoting among related entities: textbook paragraphs, sentences, concepts, and sentence encodings. • Virtuoso integration with the knowledge base.

Citation preview

Page 1: AURA Wiki - Knowledge Acquisition with a Semantic Wiki Application

AUTHORING WITH AURA WIKISemTechBiz 2013, San Francisco

… A CASE STUDY

Page 2: AURA Wiki - Knowledge Acquisition with a Semantic Wiki Application

Today we will be talking about…

• Populating a Symbolic AI – Aura• The spiraling cost structure for encoding data into a symbolic AI

• How do we bring low cost domain experts into the process?

• Creating a Semantic MediaWiki Installation• Importing a textbook into Semantic MediaWiki and marking up pages with properties

• Customizing the installation for annotating textbook sentences

Page 3: AURA Wiki - Knowledge Acquisition with a Semantic Wiki Application

AURA

Page 4: AURA Wiki - Knowledge Acquisition with a Semantic Wiki Application

3) Encoding Planning -- 35% time

Group Common UTs, ID KR/KE Issues,

ID Already Encoded, Write How to Encode

Pre-Planning, QA Check

Status Labeling: Encoding Complete, KR Issue (Closed)

2) Reaching Consensus -- 14% timeUniversal Truth Authoring, Concept Chosen QA Check

1) Determining Relevance -- 2% timeHighlighting, Diagram Analysis

QA Check

Status Labeling: Relevant, Irrelevant (Closed)

6) Question-Based Testing -- 14% timeUse Minimal Test Suite, Reasoning JIRA Issues Filed,

Encoder Fills KB GapsQA Check with Screenshots of “Passing" Comparison

and Relationship Questions

5) Key Term Review -- 25% timeKR Evaluated by Modeling Expert and Biologist,

Encoder Makes ChangesKR Evaluated by Modeling Expert and Biologist

QA Check

4) Encoding -- 10% timeEncode, File JIRA Issues

QA Check

Status Labeling: Encoding Complete, KE Issue

Page 5: AURA Wiki - Knowledge Acquisition with a Semantic Wiki Application

-- How to choose a concept given a UT?-- How to produce UTs from sentences?

Sentence

Sentence

UT

UT

UT

UT

Chapter

Chapter

KBBook

CMap

CMap

CMap

CMap

Chapter UT

2) Reaching Consensus -- 14% time

Univeral Truth Authoring, Concept Chosen

Page 6: AURA Wiki - Knowledge Acquisition with a Semantic Wiki Application

What is a Universal Truth?• “A Universal Truth is a stand-alone, unambiguous

declarative sentence about a textbook topic that expresses a single fact that is universally true”- AURA Knowledge Engineering Manual

• “Water is composed of two Hydrogen element molecules and one Oxygen element molecule with the chemical formula H20”• Water is composed of hydrogen• Water is composed of oxygen• Hydrogen is an element• Oxygen is an element• Water has the chemical formula H20

• Does: “Water is a compound” count?

Page 7: AURA Wiki - Knowledge Acquisition with a Semantic Wiki Application

Project Goals

• “Crowd Source Universal Truth Authoring”• Can Domain Experts Author Useful Universal

Truths?• Can We Speed Up Encoding a Textbook with Input

from Domain Experts?• Can We Create a UT Authoring Portal for Multiple

Textbooks?• Can Existing Social Networks Provide Domain

Experts Capable of UT Authoring?• Could Gamification be Applied to An Existing Portal

to Add Non-Domain Experts?

Page 8: AURA Wiki - Knowledge Acquisition with a Semantic Wiki Application

About the Domain Experts

• Students attending University of Washington or recent graduates

• All have a background in biology or life sciences• Native English speakers with excellent writing skills

• Each student read the chapters in question and was provided with an iPad running the Inquire application

• Students were paid for their time

Page 9: AURA Wiki - Knowledge Acquisition with a Semantic Wiki Application

A Semantic MediaWiki Portal

Page 10: AURA Wiki - Knowledge Acquisition with a Semantic Wiki Application

Storing a Text Book in Aura Wiki• The wiki was created with instances of page types

composed of textbook sentences• Sentence• Paragraph• Section• Chapter• Book

• The wiki also has imported resources to aid in the UT authoring process• Glossary Pages• Taxonomy Concepts• Universal Truths – Human and Machine

Page 11: AURA Wiki - Knowledge Acquisition with a Semantic Wiki Application

Navigating Aura Wiki

The text isn’t

readable

Is this the table of contents?

Why didn’t

I see this

first?Where’s the next

sentence?THIS PAGE HAS

NO FORMATTING

AND LOOKS

NOTHING LIKE

THE TEXTBOOK!

Page 12: AURA Wiki - Knowledge Acquisition with a Semantic Wiki Application

Navigating Aura Wiki

Removed

to Google

Analytics

Page 13: AURA Wiki - Knowledge Acquisition with a Semantic Wiki Application
Page 14: AURA Wiki - Knowledge Acquisition with a Semantic Wiki Application

Authoring Universal Truths

• Components : • Read Sentence• Access Sentence Context• Access Neighboring

Sentences• Check & Submit

Relevancy• Check & Submit Authoring

Status• Display Existing Universal

Truths• Author Universal Truths

Page 15: AURA Wiki - Knowledge Acquisition with a Semantic Wiki Application

Authoring Universal Truths

• Semantic Wiki Properties• Each page has a unique id

for the table of contents element

• The sentence itself is an element

• Elements pointing to the previous and next sentences.

• Elements pointing to top level entities

• Users can update the sentences relevancy and encoding status.

Sentence and Context View

Page 16: AURA Wiki - Knowledge Acquisition with a Semantic Wiki Application

Authoring Universal Truths

Input form for new UT. First two inputs are

required.

Page 17: AURA Wiki - Knowledge Acquisition with a Semantic Wiki Application

Authoring Universal Truths

• Semantic Wiki Properties• Reference sentence• The universal truth text• UT concept – AURA provided• UT context – AURA provided• Accuracy rating for the universal

truth• Date created, approved, and

when ratings were applied

Universal Truth

Page 18: AURA Wiki - Knowledge Acquisition with a Semantic Wiki Application

PROPOSALSUser Experience Review

… 5 of 5,000

Page 19: AURA Wiki - Knowledge Acquisition with a Semantic Wiki Application

Navigating Aura Wiki• Unregistered and Registered Main Pages

• Unregistered users are locked out• Registration is turned off for anonymous users

• Unique Extensions Proposed for Guided Authoring

Page 20: AURA Wiki - Knowledge Acquisition with a Semantic Wiki Application

How to View a Textbook Paragraph?Get rid of

title id

elements

Context needs to

include more

Click on

sentences in

context to

annotate next

Auto create triple format UTs from

sentence?

Page 21: AURA Wiki - Knowledge Acquisition with a Semantic Wiki Application

How to View a Universal Truth Page?

Can you edit

a UT you

didn’t create?

Is this edit another version

of the wiki page?

Is rating a UT

another

version of the

page?

How do we unify versions of the page for export

to AURA?

Page 22: AURA Wiki - Knowledge Acquisition with a Semantic Wiki Application

Knowledge Engineer Editing

Faceted

Browsing by

properties

Bulk editing by sentence /

propertyBulk

moderation of

universal

truths

Page 23: AURA Wiki - Knowledge Acquisition with a Semantic Wiki Application

Knowledge Engineer Editing

Page 24: AURA Wiki - Knowledge Acquisition with a Semantic Wiki Application

STUDENT REVIEWCan Experts Author Universal Truths?

Page 25: AURA Wiki - Knowledge Acquisition with a Semantic Wiki Application

Domain Expert Authoring Statistics• 6 University of Washington Students participated in the

test• Each received 45 minutes of training on creating

Universal Truths• Each was given 1 hour and a pre-selected list of

sentences on a user page to complete• The groups generated over 100+ Universal Truths each

session• They averaged 37 Universal Truths an hour per student• Students were frequently observed using their domain

experience to construct UTs not specifically worded in the source sentence (ie: “Water is a compound”)

Page 26: AURA Wiki - Knowledge Acquisition with a Semantic Wiki Application

CONCLUSION

Page 27: AURA Wiki - Knowledge Acquisition with a Semantic Wiki Application

Project Goals

• “Crowd Source Universal Truth Authoring”• Can Domain Experts Author Useful Universal Truths?

• Can We Speed Up Encoding a Textbook with Input from Domain Experts?

Page 28: AURA Wiki - Knowledge Acquisition with a Semantic Wiki Application

Project Goals

• “Crowd Source Universal Truth Authoring”• Can We Create a UT Authoring Portal for Multiple Textbooks?

Page 29: AURA Wiki - Knowledge Acquisition with a Semantic Wiki Application

Project Goals

• “Crowd Source Universal Truth Authoring”• Can Existing Social Networks Provide Domain Experts Capable of UT Authoring?

• Could Gamification be Applied to An Existing Portal to Add Non-Domain Experts?

Page 30: AURA Wiki - Knowledge Acquisition with a Semantic Wiki Application

QUESTIONS? COMMENTS?

Page 31: AURA Wiki - Knowledge Acquisition with a Semantic Wiki Application

THANK YOU(clap now)