Structure Learning on Large Scale Common

Sense Statistical Models of Human State

Advisor: Hsin-His ChenReporter: Chi-Hsin YuDate: 2009.02.11

From AAAI 2008

William Pentney, Department of Computer Science & Engineering University of Washington

Matthai Philipose, Intel Research Seattle

Jeff Bilmes, Department of Electrical Engineering University of Washington

Common Sense Data Acquisition for Indoor Mobile Robots◦ AAAI 2004◦ Rakesh Gupta and Mykel J. Kochenderfer

Sensor-Based Understanding of Daily Life via Large-Scale Use of Common Sense ◦ AAAI 2006◦ William Pentney, et al., Matthai Philipose (Intel)

Learning Large Scale Common Sense Models of Everyday Life ◦ AAAI 2007◦ William Pentney, et al., Matthai Philipose (Intel)

Related Papers

Introduction Data Acquisition and Representation Inference Evaluation Methodology and Results Conclusion

Outlines

Common sense◦ being critical to the automated understanding of

the world (Example)◦ OMICS (Open Mind Indoor Common Sense) project

This paper◦ enabling correspondingly large scale sensor-based

understanding of the world (RFID)

Introduction

Challenges◦ semantic gaps (facts in DB - phenomena detected by

sensors)◦ fragility of reasoning in the face of noise ◦ Incompleteness of repositories (DB) ◦ slowness of reasoning with these large repositories

The adaptation of using sensor data is challenging because it is unclear that …◦ how to represent models◦ term occurrence statistics are a practical means of

acquiring arbitrary common sense information from the web

Introduction (Cont.)

Collecting common sense data through the Open Mind Indoor Common Sense (OMICS) website

Restricting the domain to indoor home and office environments

Data Acquisition – OMICS(AAAI 2004)

Data Acquisition – OMICS (Cont.)

Hand proximity to objects implies object use. Three users perform various daily activities.

A total of 5-7 minutes of each activity was collected, for a total of 70-75min of data.

These traces were divided into time slices of 2.5s.

Data Acquisition – Sensor Data

Data Acquisition and Rep. - Flow

SRCS=State Recognition using Common Sense

Template to relation ◦ “You <eat> when you are <hungry>.” ◦ Relation: people(<Action>, <Context>)◦ Ex:

Relation People people (‘eat’, ‘hungry’) people(’drink water’,’are thirsty’)

Relation ContextAction contextactions(’full garbage bag’, ’put the garbage in’, ’trash’) contextactions(’making toasted bread’, ’slice’, ’bread’)

Relation ActionGenealization actiongeneralization(’investigate cause of’, ’alarm’, ’smoke alarm’) actiongeneralization(’wipe off’, ’floorcover’, ’carpet’) actiongeneralization(’clean’, ’floorcover’, ’carpet’)

SRCS ◦ 50000+ instances, 15 relations

KnowItAll◦ Weighting the facts in OMICS

Data Acquisition and Rep. –DB

Weighted Relations

Using 20 fixed rewrite rules◦ People(S, A) ⇝ (actionObserved(A) ⇒ personIn(S))◦ People(angry, yell) ⇝ (actionObserved(yell) ⇒

personIn(angry)) Horn clause

◦ p1∧ p2 ∧ … ∧ pN ⇒ pN+1

◦ P1 … pN : constant/atom, pN+1 :atom◦ Constants: object, action, location, context, state◦ 8 types of atoms:

useInferred(O), stateOf(O, S), locationInfererd(L), personIn(S), actionObserved(A)

Data Acquisition and Rep. –Weighted Relations

Weighted Horn Clauses

MRF◦ Consists a graph whose set of vertices V are

connected by a set of cliques ci ⊂ V. V: atom fi,t and object oi,t for all i in time t

◦ Each ci has a potential function φi:ciR+

◦ Calculates p(ft,ot)

Markov logic network (Richardson & Domingos 2006)◦ Are used for this conversion.

Data Acquisition and Rep. – Weighted

Horn Clauses

Markov Random Field

(MRF)

Data Acquisition and Rep. –Markov Random

FieldChain Graph

At time slice t◦ Useinferred(O)=true if the use of object is

detected at time slice t◦ Infer other unknown variables using loopy belief

propagation (Pearl 1988) Calculate marginal for propositions in time slice t-1

30 min for inference in each time slice◦ Use query-directed pruning to improve it

Output thresholds◦ Set a variable to true if p(variable) > thresholds◦ Use decision stump to learn the threshold

Inference

Monitoring 24 Boolean variables to identify individual activity

Human labeling of the traces as the ground truth

Evaluation Methodology and Results

Train decision stumps on a sampling of data fro each activity (20mins)

Evaluation Methodology and Results (Cont.)

Evaluation Methodology and Results (Cont.)

AAAI 2008

Evaluation Methodology and Results (Cont.)

AAAI 2006

The performance is inconsistent.

Improved

Future works◦ The cost of labeling trace is expensive

semi-supervised algorithm may help◦ Integrating other sources of sensory input is

exploring◦ Selection of variable subset is important for

scalable inference. Conclusions

◦ Densely deployable wireless sensors made things possible.

Conclusion

Thanks!!