Upload
angelina-watkins
View
218
Download
0
Embed Size (px)
DESCRIPTION
of FILL-IN Multi-Interpreter Architecture Overview 3 KB Management: Abstract types Onto dimensions Surface symbols Event Frames Inference Comparisons Manipulations Logical/Computational Interpretation Token-symbol Symbol-type role Type role-EF role Current EF Management Merge from world Merge from KB World Inputs/Outputs Text, Databases, Imagery KB Management Event frame dimensions ISA relationships Containment relationships Attributes Event roles Event models
Citation preview
Multi Interpreter Architecture Overview
of FILL-IN
Multi-Interpreter Architecture Overview
2
KB Management:• Abstract types• Onto dimensions • Surface symbols• Event Frames (EF)
Inference• Comparisons• Manipulations• Logical/Computational
Interpretation• Token-symbol• Symbol-type role• Type role-EF role
Current EF Management• Merge from world• Merge from KB
World Inputs/Outputs• Text , Databases , Imagery
of FILL-IN
Multi-Interpreter Architecture Overview
3
KB Management:• Abstract types• Onto dimensions • Surface symbols• Event Frames
Inference• Comparisons• Manipulations• Logical/Computational
Interpretation• Token-symbol• Symbol-type role• Type role-EF role
Current EF Management• Merge from world• Merge from KB
World Inputs/Outputs• Text , Databases , Imagery
KB ManagementEvent frame dimensionsISA relationshipsContainment relationshipsAttributesEvent rolesEvent models
of FILL-IN
Multi-Interpreter Architecture Overview
4
KB Management:• Abstract types• Onto dimensions • Surface symbols• Event Frames
Inference• Comparisons• Manipulations• Logical/Computational
Interpretation• Token-symbol• Symbol-type role• Type role-EF role
Current EFManagement• Merge from world• Merge from KB
World Inputs/Outputs• Text , Databases , ImageryInterpretation
Token – symbolSymbol - type rolesType roles-phys repsEvent frame - boundariesEvent frame builder
of FILL-IN
Multi-Interpreter Architecture Overview
5
KB Management:• Abstract types• Onto dimensions • Surface symbols• Event Frames
Inference• Comparisons• Manipulations• Logical/Computational
Interpretation• Token-symbol• Symbol-type role• Type role-EF role
Current EF Management• Merge from world• Merge from KB
World Inputs/Outputs• Text , Databases , ImageryInference
Event role classifiersSpatial constraintsAttribute constraintsEvent role processing
of FILL-IN
Multi-Interpreter Architecture Overview
6
KB Management:• Abstract types• Onto dimensions • Surface symbols• Event Frames
Inference• Comparisons• Manipulations• Logical/Computational
Interpretation• Token-symbol• Symbol-type role• Type role-Event Frame
role
Current Event Frame Management• Merge from world• Merge from KB
World Inputs/Outputs• Text , Databases , Imagery
Current Event Frame Management
Co-referenceBelief managementSpatio-temporal orderingEvent linking
MI Interpretation subsystem overview
Interpretation Subsystem Overview
Types and Schema
KnowledgeBase
PreprocessingChain
SymbolData
Event FrameBuilder
ResolvedEvent frames
Knowledge Base
Event frame Builder
LC SlotManipulator
SymbolData LC Slot
FillerPhysical Rep
Matcher
Best Physical Representation Fit
ResolvedEvent frames
InputDocuments
Objective: Extract semantics from sequences of word/tokens into typed expressions/event frames.
• Single phase of syntactic and semantic processing
• Very High Level language-coupled ontology
• Novel Language-Independent Logical Form, using LC-Types as the basic building blocks for both event frame classes and
instances.
NLP Pipeline Processing Chain
Sentence
Detector
Doc
umen
t Tex
t
Tokenizer
Sen
tenc
es
Multi-Token
Aggregator
Toke
ns
Annotation Result
Mul
ti-To
kens
POS Tagger
Pen
n Tr
eeba
nk
PO
S T
ags
POS Mapper
Phy
sica
l Rep
P
OS
Tag
sTempora
l Resoluti
onN
atty
Res
ults
Regex Stage(s
)In
fo U
pdat
es
NLPInterpre
t
Par
se
Res
ults
Event frame
Building
Eve
nt fr
ame
Stru
ctur
es
Knowledge Base
Preprocessing
Knowledge Base
LC Event Frame Builder Approach
Goal: Achieve understanding of what is being asserted and the associated actors.
Approach: Use semantic cues and dynamic composability of LC logical schema to resolve relationships between event frames and derive object roles within them.
Semantic cues provide functional relationship between event framesand individual components within them.
Dynamic composability of event frames allows:
• Associations of these relationships with the appropriate components
• Tracking of temporal associations within multiple event frames
Symbol Data
S1 S2 … SN
Event Frame Builder Component Description
LC Slot Manipulator Physical Representation Matcher
LC Slot Filler
InterpretationGraph
Physical RepresentationMappings
KB Terms
KnowledgeBase
KB Term
LC-Type Rules• Fit symbol into L/C
slot of existing or new Event frame
Template Matching• Use grammatical
cues to switch between event frames
Finite State Machine Processing
012
01
Phys Rep 1
Phys Rep 2
S1
S2
S3
S4
KeysElement Symbol
Example of MI processing a sentence from one of the
motivating scenario sentences
Event Frame Builder Example(Initial Event Frame Boundaries)
“021715zmay14 imagery shows one SA-6 tel with three missiles loaded, one straight flush fire control radar, and two support vehicles ivo 373331n 1143335w.”
Found three LC-type expressions (i.e., event frames):
1) First-Order Expression: “0211715zmay 14 imagery shows”
• Captures an assertion about the “image”, (i.e., “image” is the Locator, “shows” is the associated Process captured as Content)
2) Second-Order Expression: “one SA-6 tel with three missiles loaded, one straight flush fire control radar, and two support vehicles ivo 373331n 1143335w”
• Captures association between two expressions, (i.e., a message described by a second-order expression is associated with the first-order expression). This composability enables further inferencing: What did imagery show?
3) Nested Expression: “loaded”
• Captures the content describing the locator “missiles”. This composability enables inferencing on weapon readiness.
“021715zmay14 imagery shows one SA-6 tel with three missiles loaded, one straight flush fire control radar, and two support vehicles ivo 373331n 1143335w.”
Event Frame Builder Example(Physical Representation)
Each expression (i.e., event frame) is described by a physical representation Within each expression, semantic cues, each described by its own physical representation, provide functional relationship between
expressions and individual components within expressions.• “021715zmay14” is a temporal component indicating the time of the assertion• “shows” is a second-order marker signifying functional relationship between two expressions• “with” and “ivo” are semantic cues indicating functional connection between individual components within the same
expression.
First-Order Event frame Second-Order Event frame Nested Event frame
“021715zmay14 imagery shows one SA-6 tel with three missiles loaded, one straight flush fire control radar, and two support vehicles ivo 373331n 1143335w.”
Event Frame Builder Example
An event frame Structure EF is graph representation of NLP results enabling reasoning and inferencing over data in time and space.• Logical Schemas associated with physical representations, provided by the NLP results, are used to populate the EF• Dynamic composability of LC logical schema allows tracking of temporal associations within multiple expressions.• Events are represented by a number of slots enabling tracking of associated attributes, time and space per expression basis.
The start and end of theevent is the same
The space of theevent is unknown No other ACTED_UPON
at completion of event
ACTOR Object or Event filling the role of ACTOR in an assertion. S Space in which event occurred
ACTION Action identifying the event S0, S1 Space at the beginning and end of event
ACTED_UPON Object or Event filling the role of ACTED_UPON in an assertion T Time during which event occurred
ACTED_UPON_T1 Object or Event filling the role of ACTED_UPON at completion of event T0, T1 Time at the beginning and end of event
“021715zmay14 imagery shows one SA-6 tel with three missiles loaded, one straight flush fire control radar, and two support vehicles ivo 373331n 1143335w.”
Event Frame Builder Example continued
“show” is a transformation event and a semantic cue for a second-order expression. The assertion fills in the “Message” slot for the “show” event.
The geolocation is realized into latitude/longitude and captured as space in the assertion.
“with” is a semantic cue for containment. There is a containment relationship between “SA-6 tel” and “three missiles”.
Example of ontological ISA information stored
in the KB
Inference using the knowledge base
18
An SA-6 is an inherited type from ground-to-air
threat
Example of user display
ReCOURSE planning alert interface
20
Detailed information
Selected JIPTL target/mission
Alerts generated against JIPTL
Drill down into alert shows source documents
21
Examples of uniformity of representations using abstract
dimensional underpinnings
of 110
Dimension labels vs. Values vs. operators vs. meta-operators
Color.green = Get_color(Object.ball) Integer.9 = Sum(Int.7, Int.2) Profit{$}.1000 = Avg((Obj.Shoes, Profit) , (Obj.Socks, Profit)) Ops_assess_score.0.9 = Assess(Plan_id.72, Phase.post_exe) Author_name.Scott = Get_author_name(Book.Waverly) Space.SW = Get_Area_of_Operations(Plan_spec.Alpha) Relevance metric.Village Sentiment =
Get_Worst performance(Relevance Metric(Plan_spec.Alpha))
Anatomy of a dimensional expression
23
Examples of logical markers
of 110
1. Primary locator (e.g., argument) markers1. Grammatical2. Locations 3. Objects 4. Attributes5. Processes
2. Primary content (e.g., predicate) markers1. Auxiliary verbs2. Processes
3. Nested expression markers4. Object role in process markers5. Function role component markers6. Second order expression markers7. Inter-expression expression markers
Expression Slot Manipulation Markers
25
26
Markerso The, A, An when found at the beginning of a sentence, or at the beginning of a clause
whose prior clause ended with a semi colon, indicate that the following substantive types are locators -and include the primary locator- until a new marker is found. Note that content markers are shown in green. Underlined text represents what is assigned the locator role. Purple text represents nested expression markers
The ball is green The old man who lives by the sea went diving A strange smell began wafting over the arena A human skull believed to be over 6000 years old was found frozen under the
ice An apple a day keeps the doctor away An apparition, like none he’d seen before was right there staring in his eyes.
of 110
Nested Expression Markers
27
Nested expression markers: the nested expression serves as a locator o Who, that, when found following one or more substantive types that followed a locator
marker and before the appearance of a content marker indicate that the following substantive types are secondary locators (i.e. they restrict the range of or serve as contents relative to the primary locator found before the marker) until the next schema role marker is found
The old man who lives by the sea is wise The person who stole my pen is in big trouble Those kids who were throwing rocks at the police were later arrested The city that never sleeps is hosting this year’s conference for persons that can’t
stay awake. The fighter that would never quit finally did Those clandestine organizations that support hacking are finally coming under
scrutiny The old car that rattled was last seen speeding down the interstate Bob, that rascal, did it again
Bungee jumping, that crazy activity, is finally gaining in popularity
Visual examples of composite event
frames
2 objects, 1 process: Asymmetric, non-transitive, non-touch to touch,
Event frame – pre execution
Event frame post execution
Event frame during execution
Obj1 Obj2
Obj1 Obj2
Obj1 Obj2
T.Pre , S.n
T.during , S.n
T.post , S.n
Pierre kissed maryPierre called maryThe tree fell on the carThe book fell on the floorPierre ran to the waterPierre stood on the ladder
Process.n
Process.n
Process.n
2 objects, 1 process: Symmetric, non-transitive, non-touch to touch,
event frame – pre execution
Event framepost execution
Event Frameduring execution
Obj1 Obj2
Obj1 Obj2
Obj1 Obj2
T.Pre , S.n
T.during , S.n
T.post , S.n
Pierre danced with MaryPierre and Mary met at the cafe
Process.n
Process.n
Process.n
3 objects, 1 Process: Symmetric, non-transitive, non-touch to touch,
Event frame– pre execution
event frame post execution
event frame during execution
Obj1 Obj2
Obj1 Obj2
Obj1 Obj2
T.Pre , S.n
T.during , S.n
T.post , S.n
Pierre and Mary shared the last cookie Pierre and Mary had a conversation
about mathPierre and Mary bought a house togetherPierre and Mary were having tea
in the treehousePierre and Mary shared the only blanket
Process.n
Process.n
Process.n
Obj3
Obj3
Obj3
3 objects where the active object 1.1 is a part of object 1, AND object 1.1 remains a part of object one during the course of executing the event frame
1 Process: Asymmetric, non-transitive, non-touch to touch, Typically object 1 = argument and the rest = predicate
event frame – pre execution
event frame post execution
event frame during execution
Obj3
Obj3
Obj3
T.Pre , S.n
T.during , S.n
T.post , S.n
Pierre kissed mary with his lipsPierre slapped the officer with his handPierre pounded the door with his fistThe tree trunk fell on the roof The book hit the floor with its spinePierre put his foot in the waterPierre gripped the ladder with his toes
Process.n
Process.n
Process.n
Obj1Obj1.1
Obj1Obj1.1
Obj1Obj1.1
3 objects where the active object 1.1 is associated with object 1 pre-execution, AND object 1.1 disassociates from object one during the course of executing the event frame
1 Process: Asymmetric, non-transitive, non-touch to touch, Typically object 1 = locator and the rest = content
event frame – pre execution
event frame post execution
event frame during execution
Obj3
Obj3
Obj3
T.Pre , S.n
T.during , S.n
T.post , S.n
Pierre gave his shirt to the victimPierre used his shirt to trap the mouseThe huge branch broke off [the tree] and fell on the roof The book jacket blew off and hit the floor The plane took off from the carrier to look for survivorsPierre’s hat fell off and hit the ground.
Process.n
Process.n
Process.n
Obj1Obj1.1
Obj1Obj1.1
Obj1Obj1.1
3 objects where object 1 is touching an independent object 2 in such a way that object 2 then touches object 3.
1 Process: Asymmetric, non-transitive, non-touch to touch, Typically object 1 = locator and the rest = content
event frame – pre execution
event frame post execution
event frame during execution
Obj3
Obj3
Obj3
T.Pre , S.n
T.during , S.n
T.post , S.n
Pierre surrendered his gun to the officer Pierre gave the book to maryPierre handed the papers to the undercover copPierre sold the car to his neighborPierre gave the sandwich to his brother
Process.n
Process.n
Process.n
Obj1 Obj2
Obj1Obj2
Obj1Obj2
4 objects where object 1 is touching an independent object 2 in such a way that object 2 then touches a part (object 3.1) of object 3.
1 Process: Asymmetric, non-transitive, non-touch to touch, Typically object 1 = locator and the rest = content
event frame – pre execution
event frame post execution
event frame during execution
T.Pre , S.n
T.during , S.n
T.post , S.n
Pierre put the gun in the officer’s hand Pierre placed the book on mary’s headPierre landed his plane on the deck of the carrierPierre threw the peanuts in to the elephant’s mouth
Process.n
Process.n
Process.n
Obj1 Obj3Obj3.1
Obj1Obj2
Obj1Obj2
Obj3Obj3.1
Obj3Obj3.1
Obj2
Obj2
5 objects where obj 1.1, a part of object 1, is touching an independent object 2 in such a way that object 2 then touches a part (object 3.1) of object 3.
1 Process: Asymmetric, non-transitive, non-touch to touch, Typically object 1 = locator and the rest = content
event frame – pre execution
event frame post execution
event frame during execution
T.Pre , S.n
T.during , S.n
T.post , S.n
Pierre used his elbows to put the gun in the officer’s hand Using only his pinkies, Pierre placed the book on mary’s headPierre landed feet first on the deck of the carrierPierre used his good hand to throw
peanuts in to the elephant’s mouth
Process.n
Process.n
Process.n
Obj1 Obj3Obj3.1
Obj1Obj2
Obj1Obj2
Obj3Obj3.1
Obj3Obj3.1
Obj1.1
Obj1.1
Obj1.1