Knowledge Processing for Autonomous Robots

Knowledge Processing for AutonomousRobots

Moritz Tenorth

Intelligent Autonomous Systems GroupTechnische Universitat Munchen

November 5, 2010

Motivation Prolog KnowRob Query Extend Integrate Use Outlook

Outline

1. Why Knowledge Processing for Autonomous Robots?

2. Introduction to Prolog

3. Overview of KnowRob

4. Asking Queries

5. Extending KnowRob

6. Integrating Perception Components

7. Inferring Control Decisions with KnowRob

8. Outlook

November 5, 2010

Knowledge Processing for Autonomous Robots Moritz Tenorth


Outline




4. Asking Queries




8. Outlook

November 5, 2010



Knowing what you are looking at

I Can you eat it or drink it?

I Is it perishable?

I Is it fragile?

I Can you recognize it?

I Where would you put it?

I How much is it?

I Have you seen anythingsimilar before?

November 5, 2010



Knowing what you are looking at

Not only labels, butI Sub-class relations: electrical

appliance, household item, coolingdevice

I properties of objects: fragile,heavy, perishable, edible

I relations to other objects: storagelocation, similarity

I integration of knowledge sources:common sense, environment,perception, human dialog

November 5, 2010



Knowing where to look for something

I Basic:Fixed controller for opening everything with a handle

I Parameterizable:Look up the positions of all containers in the map

I Knowledge-based:Infer the right container based on the types of the object andthe container

November 5, 2010







November 5, 2010







November 5, 2010




storagePlaceFor

instanceOf

instanceOf

November 5, 2010



Outline




4. Asking Queries




8. Outlook

November 5, 2010



Why Prolog for autonomous robots?

I Can be read declaratively

I Can be programmed (prodecurally)

I There are textbooks and tutorials

I Industrial-strength open-source implementations

I Foreign language interfaces

I Result interpretation: similar to a DB query language

November 5, 2010



Knowledge base for the milk example

jmilk.plj

cup(cup0).

dairyProduct(milk1).

meatProduct(ham2).

cupboard(cupboard3).

refrigerator(fridge4).

perishable(Prod) :- dairyProduct(Prod); meatProduct(Prod).

storagePlaceFor(Loc, Item) :- refrigerator(Loc), perishable(Item).

storagePlaceFor(Loc, Item) :- cupboard(Loc), cup(Item).

searchForIn(Item,Loc) :- storagePlaceFor(Loc, Item).

November 5, 2010



Example queries

Two main types of queries:

I Query with unbound variables → variable bindings

I All variables bound → true/false result

Example queries

$ prolog -f milk.pl

?- refrigerator(A).

A = fridge4.

?- refrigerator(fridge4).

true.

?- searchForIn(milk1, A).

A = fridge4 .

November 5, 2010



Example queries

Example queries

?- storagePlaceFor(A, milk1).

A = fridge4 .

?- storagePlaceFor(fridge4, A).

A = milk1 ;

A = ham2 ;

?- storagePlaceFor(A, B).

A = fridge4,

B = milk1 ;

A = fridge4,

B = ham2 ;

A = cupboard3,

B = cup0.

November 5, 2010



Under the hood: the tracer

Tracing a query

?- trace, searchForIn(milk1, P).

Call: (7) searchForIn(milk1, G369) ? creep

jmilk.pljsearchForIn(Item, Loc) :- storagePlaceFor(Loc, Item).

November 5, 2010




Tracing a query



. Call: (8) storagePlaceFor( G369, milk1) ? creep

jmilk.pljstoragePlaceFor(Loc, Item) :- refrigerator(Loc), perishable(Item).


November 5, 2010




Tracing a query




. . Call: (9) refrigerator( G369) ? creep

jmilk.pljrefrigerator(fridge4).

November 5, 2010




Tracing a query





. . Exit: (9) refrigerator(fridge4) ? creep

jmilk.pljrefrigerator(fridge4).

November 5, 2010




Tracing a query






. . Call: (9) perishable(milk1) ? creep

jmilk.pljdairyProduct(milk1).


November 5, 2010




Tracing a query







. . . Call: (10) dairyProduct(milk1) ? creep



November 5, 2010




Tracing a query








. . . Exit: (10) dairyProduct(milk1) ? creep



November 5, 2010




Tracing a query









. . Exit: (9) perishable(milk1) ? creep



November 5, 2010




Tracing a query










. Exit: (8) storagePlaceFor(fridge4, milk1) ? creep

jmilk.pljstoragePlaceFor(Loc, Item) :- refrigerator(Loc), perishable(Item).


November 5, 2010




Tracing a query











Exit: (7) searchForIn(milk1, fridge4) ? creep


November 5, 2010




Tracing a query











Exit: (7) searchForIn(milk1, fridge4) ? creep

P = fridge4 .


November 5, 2010



Prolog syntax

I Predicate names and constants start with a lowercase letter orare inside single quotes

I Variables start with an uppercase letter

I Formulas end with a full stop ’.’

I Stepping through all results of a query with ’;’

I Logical operators:

Operator Prolog equivalent Example∧ , type(A, ’Cup’), on(A, T).∨ ; type(A, ’Food’); type(A, ’Drink’).

jExamplej

dairyproduct(milk).

perishable(Stuff) :- storedIn(Stuff, Ref), refrigerator(Ref).

November 5, 2010



Outline




4. Asking Queries




8. Outlook

November 5, 2010



The KnowRob knowledge base

Machine learning

Clustering,Classification,Segmentation

KNOW-ROB

Reasoners

DL, MLN, BLN

Natural languageprocessing

Physicalreasoning

Action-relatedconcepts

Actionabstraction

Encyclopedicknowledge

Common-senseknowledge

Queryinterface

Visualizationmodules

ROS SQL

human

Computable classes and properties

map vision human human

November 5, 2010



Launching KnowRob

I KnowRob is very modular: Dependencies are automaticallyloaded by rosprolog

I General syntax: rosrun rosprolog rosprolog <pkg>I starts a Prolog environmentI checks for dependencies of <pkg>I locates them on the diskI loads the prolog/init.pl file for each of these packages

I For today we use:rosrun rosprolog rosprolog knowrob tutorial

November 5, 2010



Outline




4. Asking Queries




8. Outlook

November 5, 2010



Querying KnowRob from your program

I Tell/ask interface via a ROS serviceI tell: add statementsI ask: send a query

I Send queries as string like you would do from a console

I Result serialized using json

I Client libraries for C++, Python, Java, Lisp available

November 5, 2010



Querying KnowRob from Python

jknowrob/json prolog/examples/test json prolog.pyj

#/usr/bin/env python

import roslib; roslib.load manifest(’json prolog’)

import rospy

import json prolog

if name == ’ main ’:

rospy.init node(’test json prolog’)

prolog = json prolog.Prolog()

query = prolog.query("member(A, [1, 2, 3, 4]), B = [’x’, A]")

for sol in query.solutions():

print ’Found solution. A = %s, B = %s’ % (sol[’A’], sol[’B’])

query.finish()

November 5, 2010



Querying KnowRob from C++

jknowrob/json prolog/examples/test json prolog.cppj

#include <string>

#include <iostream>

#include <ros/ros.h>

#include <json prolog/prolog.h>

using namespace std;

using namespace json prolog;

int main(int argc, char *argv[]) {

ros::init(argc, argv, "test json prolog");

Prolog pl;

PrologQueryProxy bdgs = pl.query("member(A, [1, 2, 3]), B = [’x’, A]");

for(PrologQueryProxy::iterator it=bdgs.begin(); it != bdgs.end(); it++) {

PrologBindings bdg = *it;

cout << "Found solution: " << (bool)(it == bdgs.end()) << endl;

cout << "A = "<< bdg["A"] << endl;

cout << "B = " << bdg["B"] << endl;

}return 0;

}

November 5, 2010



The KnowRob Taxonomy

I Defines the basic concepts of the household and roboticdomain

I Hierarchy of classes and their properties

I Stored in the Web Ontology Language OWL as (S,P,O) triples

November 5, 2010



The KnowRob Taxonomy

November 5, 2010



Scaling up

November 5, 2010



Class Restrictions

Asserted vs. inferred class based on restrictions

November 5, 2010



TBOX

I The TBOX contains terminological statements: abstractdescriptions of classes and properties

I owl subclass of(Sub, Super). reads all classes derived from Super

I rdfs subproperty of(SubProp, Prop). lists sub-properties of Prop

I owl individual of(Restr, owl:’Restriction’), owl subclass of(Class,

Restr). reads all restrictions that are defined on Class

I owl satisfies restriction(Res, Restr). verifies whether Ressatisfies the restriction Restr

I owl description(RestrOWL, RestrProlog). translates the OWLidentifier of a restriction into a Prolog term

November 5, 2010



Example queries: TBOX

Example

$ rosrun rosprolog rosprolog knowrob tutorial

?- owl subclass of(A, knowrob:’FoodOrDrink’).

A = ’http://ias.cs.tum.edu/kb/knowrob.owl#FoodOrDrink’ ;

A = ’http://ias.cs.tum.edu/kb/knowrob.owl#Drink’ ;

A = ’http://ias.cs.tum.edu/kb/knowrob.owl#Coffee-Beverage’ ;

A = ’http://ias.cs.tum.edu/kb/knowrob.owl#InfusionDrink’ ;

A = ’http://ias.cs.tum.edu/kb/knowrob.owl#Tea-Beverage’ ;

A = ’http://ias.cs.tum.edu/kb/knowrob.owl#Tea-Iced’

November 5, 2010



ABOX

I The ABOX contains assertional statements: concreteinstances of the abstract classes and relations between them

I owl individual of(Ind, Class) is true if Ind is an instance of Class

I owl has(S, P, O) is a general query predicate for relations Pbetween a subject S and an object O

Important: Most predicates can be used with differentcombinations of free and bound variables:

I owl individual of(Ind, knowrob:’Cup’) reads all instances of classCup

I owl individual of(knowrob:’cup0’, Class) reads all classes cup0belongs to

November 5, 2010



Example queries: ABOX

Example

?- owl has(A, rdf:type, knowrob:’Cupboard’).

A = ’http://ias.cs.tum.edu/kb/knowrob.owl#Drawer1’ ;

A = ’http://ias.cs.tum.edu/kb/knowrob.owl#Drawer103’ ;

?- owl has(knowrob:’Drawer1’, P, O).

P = ’http://www.w3.org/1999/02/22-rdf-syntax-ns#type’,

O = ’http://ias.cs.tum.edu/kb/knowrob.owl#Drawer’ ;

P = ’http://ias.cs.tum.edu/kb/knowrob.owl#widthOfObject’,

O = literal(type(xsd:float, ’0.58045006’)) ;

P = ’http://ias.cs.tum.edu/kb/knowrob.owl#properPhysicalParts’,

O = ’http://ias.cs.tum.edu/kb/knowrob.owl#Door4’ ;

November 5, 2010



Visualizing knowledge

I A java-based visualization canvas can display objects and theirlocations

I visualization canvas(C). starts the canvas and stores a handle inthe variable C

I $C refers to the last binding of the variable C

I add object(Identifier, $C). sends an object instance to thecanvas in order to visualize it

I API documentation: http://ias.in.tum.de/kb/api/mod vis.pl.html

November 5, 2010



Example queries: Visualization

Example

?- owl has(A, rdf:type, knowrob:’Cupboard’),

add object with children(A, $C).

A = ’http://ias.cs.tum.edu/kb/ias semantic map.owl#closet0’ ;

A = ’http://ias.cs.tum.edu/kb/ias semantic map.owl#cupboard0’ ;

November 5, 2010



Exercise: Querying KnowRob

Exercise sheet – Querying the Knowledge Base

November 5, 2010



Optional queries

I Sub-classes of CookingUtensil

I Everything that has a handle

I What to use for heating food?

I What needs electricity?

I What needs to be put into the fridge?(hint: predicate storagePlaceFor)

November 5, 2010



Outline




4. Asking Queries




8. Outlook

November 5, 2010



Editing Ontologies with Protege

I Graphical editor for OWL ontologies

I Download and tutorials available at http://protege.stanford.edu/

November 5, 2010



Exercise: Adding knowledge to KnowRob

Exercise sheet – Extending the Knowledge Base

November 5, 2010



Outline




4. Asking Queries




8. Outlook

November 5, 2010



Reasoning on robot data

I Robot knowledge processing = dynamic knowledge processing

I Knowledge base needs to make decisions based on the mostcurrent information

Use the robot’s data structures as extended knowledge base!

I Create instances on demand (e.g. recognized objects)

I Compute properties of and relations between them

November 5, 2010



Computables: Computation On Demand

Computables: Attachcomputational methods tosemantic relations

Two components:

I Prolog predicate tocompute the relation

I OWL file to link thepredicate to the OWLrelation

November 5, 2010



Calling computables

I Take computables into account only when needed

I Special query predicates: rdf triple, rdfs instance of

I return all normal solutions (owl has, owl individual of)I search for all computables for the class/relationI return all results of all computables

Useful predicates

rdf triple(+Pred, ?Subj, ?Obj)

rdfs instance of(?Inst, ?Class)

Note the different order of arguments for rdf triple

November 5, 2010



Task 1: Retrieve object information

I Java client to interface with ROS:src/edu/tum/cs/ias/knowrob/tutorial/ROSClient.java

I Use JPL to call the object detection service from Prolog/object detection

Useful predicates

jpl new(+Class, +Arguments, -Value)

jpl call(+Class, +Method, +Args, -Result)

jpl call(+Object, +Method, +Args, -Result)

jpl get(+Object, +Field, -Value)

jpl array to list(+JArray, -PlList)

See also: http://www.swi-prolog.org/packages/jpl/prolog api/api.html

November 5, 2010



Task 1: Retrieve object information

I Extract object information from the TabletopDetectionResult

I Pass these values to the predicates for creating the objectinstance, the perception instance, and for linking those

Useful predicates

create perception instance(-Perception)

create object instance(+ObjTypes, +ID, -ObjInst)

set object perception(+ObjInst, +Perception)

set perception pose(+Perception, [+M00, +M01, +M02, +M03,

+M10, +M11, +M12, +M13,

+M20, +M21, +M22, +M23,

+M30, +M31, +M32, +M33])

November 5, 2010



Task 1: Object representation

VisualPerception-42objectActedOn: icetea2eventOccursAt: homography9startTime: timep_1.572

KNOWROBhomography9

1 0 0 2.560 1 0 1.320 0 1 0.930 0 0 1

compCoptarget:VisualPerception

TIM

E

CoP

homography8

1 0 0 2.340 1 0 1.750 0 1 0.730 0 0 1

icetea2type: TetraPakwidth: 0.1depth: 0.1height: 0.3




homography3

1 0 0 3.040 1 0 2.250 0 1 0.930 0 0 1

homography5

1 0 0 0.940 1 0 1.050 0 1 1.530 0 0 1

November 5, 2010



Task 2: Reason about the perceived objects

Have a look at the KnowRob ontology and formulate the followingqueries on the just perceived set of objects:

I Everything that can contain other things

I All food items

I Perishable things

I The positions (x,y,z) of all drinks

Useful predicates

rdfs instance of(A, knowrob:’HumanScaleObject’)

owl individual of(Inst, Class)

owl subclass of(Sub, Super)

rdf triple(knowrob:orientation, Obj, Or)

rdf triple(knowrob:m03, Or, X)

November 5, 2010



Optional tasks

I Query for items that belong into the fridge

I Determine the pose of the handle of the storage place for alldetected perishable items

I Visualize your results in mod vis

I Write a program in Python/C++ that sends the abovequeries via ROS

I Have a look at owl/tabletop obj.owl to see how the computableis realized

November 5, 2010



Outline




4. Asking Queries




8. Outlook

November 5, 2010



Integration into the executive

Lorenz

November 5, 2010



Outline




4. Asking Queries




8. Outlook

November 5, 2010



Reasoning on objects

jcomp copj

Listen to topics and add theinformation to the knowledge base

jcomp spatial, comp temporalj

Compute qualitative spatial andtemporal relations (in, on, during)

November 5, 2010



Reasoning on actions

jcomp ehowj

Translate natural-languageinstructions into OWLrepresentations and robot plans

jcomp fipm, mod hamj

Segment and abstract humanmocap data

November 5, 2010



Reasoning

jsrldb, mod probcogj

Probabilistic inference usingMarkov Logic Networks andBayesian Logic Networks

jias prolog addons/classifiers.plj

Weka and Mallet libraries forclassifiers and clusterers

November 5, 2010



Reasoning

jmod similarityj

Compute the semantic similaritybetween concepts in the ontology

jias naive physics simj

Naive physics reasoning forcomputing relations in theknowledge base

November 5, 2010



Thank you for your attention

Questions?

PR2 Illustration by Josh Ellingson, Willow Garage

November 5, 2010


Documents

Knowledge Processing for Autonomous Robots