semantic construction grammar

+

how can NIPS help with deep reading

michael @witbrock

WHAT’S THE POINT OF CYC

(Doug Lenat)

3-Stage Research Program for AI

1. Slowly hand-code a very large and very broad KB.

2. When enough knowledge is present, the system should actively help with the KA process. It should be faster to acquire more from texts, databases, [websites], interactive dialogues, etc.

3. To go beyond the frontier of human knowledge, the system will have to rely on learning by discovery, to expand its KB domain by domain.

CYC KNOWLEDGE BASE

Thing

Universe

isa

isa

Celestial Body

isa

located in

Planet

subclass

Earth

isa

Animal

isa

Human

subclas

s

Physics

Money

Mathematics

Chemistry

Time

LearningFood

Vehicles

EventEducation

School

Language

LoveEmotions Going for a

walk

Death

Cat

Euro

Working

Words

DrivingRainStabbing someone

Nature

Tree

Hatred

Fear

Physics

Time

LearningVehicles

EventEducation

School

Emotions

Going for a walk

Death

Cat

EuroWords

DrivingRain

Stabbing someone

Nature

Tree

HatredFear

Planet

Earth

isaHuman

Physics

Money

Mathematics

Chemistry

Time

Learning

FoodVehicles

Event

EducationLanguage

LoveEmotions Going for a

walk

Cat

Euro

Working

Words

DrivingRain

Tree

HatredFear

LearningVehicles

Event

EducationSchool

Emotions

Euro

Driving

Stabbing someone

Hatred

Fear

Cycorp © 2006

Thing

Intangible

ThingIndividual

Temporal

Thing

Spatial

Thing

Partially

Tangible

Thing

Paths

Sets

Relations

Logic

Math

Human

Artifacts

Social

Relations,

Culture

Human

Anatomy &

Physiology

Emotion

Perception

Belief

Human

Behavior &

Actions

Products

Devices

Conceptual

Works

Vehicles

Buildings

Weapons

Mechanical

& Electrical

Devices

Software

Literature

Works of Art

Language

Agent

Organizations

Organizational

Actions

Organizational

Plans

Types of

Organizations

Human

Organizations

Nations

Governments

Geo-Politics

Business,

Military

Organizations

Law

Business &

Commerce

Politics

Warfare

Professions

Occupations

Purchasing

Shopping

Travel

Communication

Transportation

& Logistics

Social

Activities

Everyday

Living

Sports

Recreation

Entertainment

Artifacts

Movement

State Change

Dynamics

Materials

Parts

Statics

Physical

Agents

Borders

Geometry

Events

Scripts

Spatial

Paths

Actors

Actions

Plans

Goals

Time

Agents

Space

Physical

Objects

Human

Beings

Organ-

ization

Human

Activities

Living

Things

Social

Behavior

Life

Forms

Animals

Plants

Ecology

Natural

Geography

Earth &

Solar System

Political

Geography

Weather

General Knowledge about Various Domains

Specific data, facts, and observations

Very specific information(some indirect, via SKSI)

UpperOntology

CoreTheories

Domain-SpecificTheories

EVENT TEMPORAL-THING PARTIALLY-TANGIBLE-THING

( a, b ) a EVENT b EVENT causes( a, b ) precedes( a, b )

( m, a ) m MAMMAL a ANTHRAX • causes( exposed-to( m, a ), infected-by( m, a ) )

(ist FtLaudHolyCrossERCase#403921(caused CutaneousAnthrax

(SkinLesions Ahmed_al-Haznawit)))

First Order Predicate Calculus: unambiguous; enable mechanical reasoning

Every NZr has a Queen.Every NZr has a mother.

y.x. NZr(x) ruler(x,y)&Queen(y)x.y. NZr(x) mother(x,y)

Higher Order Logic: contexts, predicates as variables, nested modals, reflection,…

In Mt: MolecularBiologyMt

(implies(isa ?MOLECULE-TYPE TranscriptionFactor)(behaviorCapable ?MOLECULE-TYPE

(ChemicalBindingEventTypeWithTypesFn TranscriptionFactor DNAMolecule)objectOfAttachment))

(isa AVPR2-Human-GIS GeneTypeBySpeciesAndProductFamily)

(gene-GISTypeCodesForType AVPR2-Human-GIS VasopressinV2Receptor)

In Mt : VertebratePhysiologyMt

(relationExistsAll outputsCreated IntramembranousBoneGrowthAndDevelopment FlatBone)

(implies (and (isa ?MUT GeneticMutationEvent-Cellular)(locusOfCellularProcess-Cell ?MUT ?ANCESTOR)(isa ?ANCESTOR Cell)(subEvents ?REPRO ?MUT))

(abnormal (TheList ?REPRO ?PROGENY ?ANCESTOR)(implies(and

(isa ?REPRO AsexualReproductionEvent)(outputsCreated ?REPRO ?PROGENY)(isa ?PROGENY BiologicalLivingObject)(doneBy ?REPRO ?ANCESTOR))

(geneticallyIdentical ?ANCESTOR ?PROGENY))))

First Order

With Context

Rule

Exceptions

Transcription factors can bind with DNA.

Normally, the progeny of asexual reproduction are genetically

identical to the parent; however, if the parent is a cell in which a mutation has occurred, this rule

doesn’t apply.

Each vertebrate flat bone was created by

intramembranous bone development.

NIPS TASK 1: DISTRIBUTED REP OF THIS

#$TransportationEvent

#$ControllingATransportationDevice

#$TransportWithMotorizedLandVehicle

(#$SteeringFn #$RoadVehicle)

#$TransporterCrashEvent

#$VehicleAccident

#$CarAccident

#$Colliding

#$IncurringDamage

#$TippingOver

#$Navigating

#$EnteringAVehicle …

Can it be removed by pulling, if enough force is used, without

damaging either object?

– No -- Try #$in-Snugly

or #$screwedIn

Is it attached to the inside of the outer object?

– Yes -- Try#$connectedToInside

Does the inner objectstick into the outer object?

–Yes – Try#$sticksInto

Cycorp © 2007

Does part of the inner objectstick out of the container?

◦ None of it.

#$in-ContCompletely

◦ Yes

#$in-ContPartially

◦ No

• #$in-ContClosed

◦ If the container were turned around could the contained object fall out?

Yes

#$in-ContOpen

NIPS TASK 2: LEARN CONCEPTS

THIS FINELY DISTINGUISED

10incyc.cyc.com

NIPS TASK 3: LEARN TO

PRODUCE COHERENT NL

FROM REPRESENTATIONS

Semantic Construction Grammar

representational context: Cyc vocabulary and syntax

12

a sad realisation and an opportunity

logical representations good for inference are

sometimes very far from natural language

… in unpredictable ways

… and they need to be right in ways NL does not

So, syntactic mapping is (pretty much) hopeless

BUT, storage is pretty much free,

… and inference is getting there

… and ILP works at least some of the time

(EBMT ∩ FrameNet ∩ Cyc) ► SCG

Renaissance Artists

Kind of TimeInterval

Noun Form: not plural Kind of Agent-Generic

Noun form

Bronze Age Farmers

(SubcollectionOfWithRelationToFn

Artist activeDuringPeriod

TheRenaissance)

(SubcollectionOfWithRelationToFn

Farmer activeDuringPeriod

TheBronzeAge)

Yellow Submarine

Sick Child

(SubcollectionWithRelationToFnSubmarine mainColorOfObjectYellow)

(SubcollectionWithRelationToFnHumanChildstateOfHealthSick)

Attributive Adjective & Noun

Supporting Lexical Assertions:

(adjSemTrans Yellow-TheWord 0 RegularAdjFrame

(mainColorOfObject :NOUN YellowColor))

(adjSemTrans Sick-TheWord 0 RegularAdjFrame

(stateOfHealth :NOUN Sick))

15

NIPS TASK 1 again

16

mana feeds a dog pepperoni and mushroom pizza

Acre

ControllingSomething

Man-TheProgram

AdultHumanMale

FeedingEvent

Feeding-

PromotingSomething

Dog

HotDog

Pepperoni EdibleMushroom

Mushroom-Broadly

(StuffFn Mushroom-Broadly)

Pizza

17

18

50 Generalisations:Agent-Underspecified BiologicalLivingObject Boundary-

Underspecified EnduringThing-Localized

EukaryoticOrganism FiniteSpatialThing

FixedFunctionalSystem FunctionalSystem Fungus FungusBLO

Heterotroph Individual Landmark-Underspecified Location-

Underspecified Mushroom-Broadly NaturalTangibleStuff Opaque

OrganicMaterial Organism-Whole PartiallyTangible

PolyDimensionalThing PositiveDimensionalThing Region-

Underspecified SomethingExisting SpatialThing SpatialThing-

Localized SpatialThing-NonSituational SpatiallyBoundedThing

SpatiallyContinuousThing SurfaceRegion-Underspecified System

TemporalThing TemporallyContinuousThing

TemporallyExistingThing Thing ThreeDimensionalThing Trajector-

Underspecified TrueFungus (CollectionDifferenceFn

SpatialThing Situation) (CollectionUnionFn (TheSet

BiologicalLivingObject FictionalCharacter)) (CollectionUnionFn

(TheSet BiologicalLivingObject (GroupFn BiologicalLivingObject)))

(CollectionUnionFn (TheSet Fungus Bacterium)) (CollectionUnionFn

(TheSet Organism-Whole Cell)) (CollectionUnionFn (TheSet

SpatialThing VectorInterval)) (CollectionUnionFn (TheSet

TemporalThing Collection)) (MeaningInSystemFn SENSUS-

Information1997 "DECOMPOSABLE-OBJECT")

75 Generalisations:Agent-Underspecified Artifact-Generic BiologicalLivingObject Boundary-

Underspecified BusinessRelatedThing Container-Underspecified

ContainerIndependentShapedThing CulturalThing EdibleMushroom

EdibleStuff EnduringThing-Localized EukaryoticOrganism

FiniteSpatialThing FixedFunctionalSystem Food FoodOrDrink

FoodOrDrinkOrIngredient FunctionalSystem Fungus

FungusBLO GroceryMarketCategory Heterotroph HumanScaleObject

Individual Landmark-Underspecified Location-Underspecified Mushroom-

Broadly NaturalTangibleStuff NonFluidlikeTangibleThing Opaque

OrganicMaterial Organism-Whole PartiallyTangible PartiallyTangibleProduct

PolyDimensionalThing PositiveDimensionalThing Product Region-

Underspecified SolidFood SolidTangibleThing SomethingExisting

SpatialThing SpatialThing-Localized SpatialThing-NonSituational

SpatiallyBoundedThing SpatiallyContinuousThing SurfaceRegion-

Underspecified System TemporalThing TemporallyContinuousThing

TemporallyExistingThing Thing ThreeDimensionalThing Trajector-

Underspecified TrueFungus (CollectionDifferenceFn SomethingExisting

Organization) (CollectionDifferenceFn SpatialThing Situation)

(CollectionUnionFn (TheSet Action Artifact-Generic PropositionalConceptualWork))

(CollectionUnionFn (TheSet BiologicalLivingObject FictionalCharacter))

(CollectionUnionFn (TheSet BiologicalLivingObject (GroupFn

BiologicalLivingObject))) (CollectionUnionFn (TheSet DurableGood ServiceEvent

Product)) (CollectionUnionFn (TheSet FirstOrderCollection Artifact-Generic))

(CollectionUnionFn (TheSet Fungus Bacterium)) (CollectionUnionFn (TheSet

Organism-Whole Cell)) (CollectionUnionFn (TheSet SolidTangibleThing

LiquidTangibleThing)) (CollectionUnionFn (TheSet SpatialThing VectorInterval))

(CollectionUnionFn (TheSet TemporalThing Collection)) (EdibleByFn

HomoSapiens) (MeaningInSystemFn SENSUS-Information1997

"DECOMPOSABLE-OBJECT") (MeaningInSystemFn SENSUS-Information1997

"NONDECOMPOSABLE-OBJECT") (MeaningInSystemFn SENSUS-

Information1997 "SEPARABLE-ENTITY") …

EdibleMushroomMushroom-Broadly

NIPS TASK 5: evaluate (subtype ?X ?Y) in detail

19

Dog HotDog

110 Generalisations, including:AboveGround AerobicBLO AerobicOrganism Agent-Generic

Agent-NonArtifactual Agent-NonGeographical Agent-

PartiallyTangible Agent-Underspecified AirBreathingVertebrate

Amniote Animal AnimalBLO AxisymmetricObject

BilateralObject BilaterallySymmetricObject BiologicalLivingObject

Boundary-Underspecified CanineAnimal CanisGenus Carnivore

CarnivoreOrder CellularTangibleThing ChordataPhylum

Coelomate CompositeTangibleAndIntangibleObject Container-

Underspecified ContainerIndependentShapedThing

DiploidOrganism Diplont Dog DomesticatedAnimal

EmbodiedAgent EnduringThing-Localized EukaryoticOrganism

Eutheria FiniteSpatialThing FixedFunctionalSystem

FrontAndBackSidedObject FunctionalSystem Heterotroph

HexalateralObject Homeotherm HumanScaleObject Individual

IndividualAgent Landmark-Underspecified

LeftAndRightSidedObject Location-Underspecified Mammal

MulticellularOrganism NaturalTangibleStuff

NonFluidlikeTangibleThing NonHumanAnimalNonPersonAnimal Opaque OrganicMaterial Organism-Whole

PartiallyIntangible PartiallyIntangibleIndividual PartiallyTangible

PerceptualAgent PerceptualAgent-Embodied

PolyDimensionalThing PositiveDimensionalThing Quadruped

Region-Underspecified Sentient SentientAnimal

SolidTangibleThing SomethingExisting SpatialThing SpatialThing-

Localized SpatialThing-NonSituational SpatiallyBoundedThing

SpatiallyContinuousThing SurfaceRegion-Underspecified System

TameAnimal TemporalThing TemporallyContinuousThing

TemporallyExistingThing TerrestrialFunctioningObject

TerrestrialOrganism Thing ThreeDimensionalThing

TopAndBottomSidedObject Trajector-Underspecified Vertebrate

ViviparousAnimal …

68 Generalisations, including:Agent-Underspecified Artifact Artifact-Generic Artifact-

NonAgentive Boundary-Underspecified BusinessRelatedThing

Container-Underspecified ContainerIndependentShapedThing

CulturalThing EdibleStuff EdibleStuffComposite

EnduringThing-Localized FiniteSpatialThing Food FoodComposite FoodOrDrink FoodOrDrinkOrIngredient

GroceryMarketCategory HotDog HumanScaleObject

InanimateObject InanimateObject-NonNatural Individual

InertArtifact Landmark-Underspecified Location-Underspecified

NonFluidlikeTangibleThing NonNaturalThing Opaque

OrganicMaterial PartiallyTangible PartiallyTangibleProduct

PolyDimensionalThing PositiveDimensionalThing PreparedFood

Product Region-Underspecified Sausage SolidFood

SolidFoodArtifact SolidTangibleArtifact SolidTangibleThing

SomethingExisting SpatialThing SpatialThing-Localized

SpatialThing-NonSituational SurfaceRegion-Underspecified

TemporalThing TemporallyExistingThing Thing

ThreeDimensionalThing Trajector-Underspecified

(CollectionDifferenceFn SomethingExisting Organization)

(CollectionDifferenceFn SpatialThing Situation)

(CollectionDifferenceFn TemporalThing BiologicalLivingObject)

(CollectionUnionFn (TheSet Action Artifact-Generic

PropositionalConceptualWork)) (CollectionUnionFn (TheSet

DurableGood ServiceEvent Product)) (CollectionUnionFn (TheSet

Event Artifact)) …

20

Logic: (#$equalSymbols ?PIZZA

(#$PizzaFn (#$TheSet $EdibleStuff#1 $EdibleStuff#0) ))

Var: ?PIZZA Type: #$Pizza

$EdibleStuff#0 and $EdibleStuff#1 pizza

21

Logic: (#$equalSymbols ?EVT (#$CollectionSubsetFn #$FeedingEvent (#$TheSetOf ?EVENT

(#$thereExists ?FEEDER (#$thereExists ?FED (#$thereExists ?FOOD (#$and (#$isa ?FOOD

$EdibleStuff#2) (#$consumedObject ?EVENT ?FOOD) (#$isa ?FED $Animal#1)

(#$recipientOfService ?EVENT ?FED) (#$isa ?FEEDER $IntelligentAgent#0) (#$providerOfService

?EVENT ?FEEDER))))))))

Var: ?EVT Type: #$FeedingEvent

a $IntelligentAgent#0 feeds a $Animal#1 $EdibleStuff#2

Safely

Execute

Vision for KB-based Protection

Cyc KB

and Inference

„scripted‟ attack

profile

high-level syntax heuristic(implies

(and

(programSubsumesInstruction ?PROG ?ALLOC-CALL)

(codingFunctionInvoked ?ALLOC-CALL ?ALLOC-FN)

(isa ?ALLOC-FN MemoryAllocationFunction)

(methodCallReturn ?ALLOC-CALL ?ALLOC-VAL)

(candidateNullValueInputTo ?ALLOC-VAL ?ADD-OP)

(programSubsumesInstruction ?PROG ?ADD-OP)

(arithmeticFunctionOfOperation ?ADD-OP PlusFn)

(oPHistOutputsToArgumentPosition

?ADD-OP ?ALLOC-VAL ?ARGN ?WRITE-OP)

(programSubsumesInstruction ?PROG ?WRITE-OP)

(codingFunctionInvoked ?WRITE-OP ?WRITE-FN)

(isa ?WRITE-FN DataLocationAlteringFunction)

(mutableArgumentPosition ?WRITE-FN ?ARGN))

(programHasPropensityForActionOfTypeDueToStatement

?PROG ?WRITE-OP OutOfRangePointerOffsetWrite))

low-level heuristic

Use reading

+ KA to

script

attack

profiles

Induce high-

level

heuristics

Sp

ec

ializ

e

Connect

data

Code

Analysis

DB

Reading knowledge about a code pattern

Describes

weak code

26

Understood code description is directly

usable with SPICE analyzed binaries

27

Logical Description “knows” how to use the data

Part of read spec

NIPS TASK 6: SUPPORT THIS

LEVEL (or more) OF INFERENCE

28

Read description of weakness allows its ID in code

& human-understandable justification

29

Inference During Interpretation

Plausibility reasoning

30

9 Shades of FailTerm interpretation fails lenient CycL truth test (“WFF”)

Arg required to be a collection but is not

Arg not an instance of all argument type constraints (strict)

Arg provably disjoint with a constraint (collections)

Arg is provably not-isa or not-genl a constraint (everything)

Argument is probably bad since it conflicts with implicit

argument constraints via #$relationNotExistsExists,

#$relationAllExists, or

#$someTypePlaysRoleInSituationType KB knowledge

Volume mismatch between types using

#$typicallyMoreVoluminousThan KB knowledge

Esoteric

Unlikely

31

Argument is probably bad since it conflicts with implicit

argument constraints via #$relationNotExistsExists,

#$relationAllExists, or

#$someTypePlaysRoleInSituationType KB knowledge

(#$SubcollectionOfWithRelationToTypeFn #$Fist

#$properPhysicalParts #$EthnicGroupOfRussians)

(relationAllExists #$Fist #$properPhysicalParts

#$AnimalBodyPart)

32

Volume mismatch between types

Can horses jump?: by analogy with Tahoe bars.

(#$SubcollectionOfWithRelationToTypeFn

#$SawHorse #$objectFoundInLocation #$Can)

This interpretation is blocked if we can prove:

(#$typicallyMoreVoluminousThan #$SawHorse #$Can)

33

Plausibility: Flat sheets of paper

Sheets that are part of an apartment unit that are part of some paper

(#$sentencePlausibilityScore

(#$Quote

(#$equalSymbols ?X

(#$SubcollectionOfWithRelationFromTypeFn

(#$SubcollectionOfWithRelationFromTypeFn #$BedSheet

#$physicalParts #$ApartmentUnit) #$physicalParts #$Paper)))

?SCORE) in #$PlausibilityQueryMt

→

?SCORE: (#$NumericLikelihoodFn 0.08)

34

Plausibility: “Play chamber

music”

(#$SubcollectionOfWithRelationToTypeFn

#$PlayingAMusicalInstrument #$objectActedOn #$ChamberMusic)

one can either call why-sentence-plausibility-factor:

(why-sentence-plausibility-factor

„(#$SubcollectionOfWithRelationToTypeFn

#$PlayingAMusicalInstrument #$objectActedOn #$ChamberMusic))

→

0.06

(# # (:GENLS (#$genls #$PlayingAMusicalInstrument #$PhysicalEvent)

#$PlausibilityQueryMt :TRUE-DEF) (:DISJOINTWITH (#$disjointWith

#$MusicalInstrument #$ChamberMusicComposition)

#$PlausibilityQueryMt :TRUE-DEF))

35

36

end

• Primacy of Semantics

• Importance of mapping not to

“logical form”, but to logic

• The frightening complexity of

human level knowledge

• Importance of doing inference

during understanding

• The time is now/nigh for uniting

the threads of AI

37