Learning and Reasoning for Robot Sequential Decision Making under Uncertainty

Saeid Amiri1, Mohammad Shokrolah Shirazi2, Shiqi Zhang1

1 SUNY Binghamton, Binghamton, NY 13902 USA2 University of Indianapolis, Indianapolis, IN 44227 USA

samiri1@binghamton.edu; shirazim@uindy.edu; szhang@cs.binghamton.edu

AbstractRobots frequently face complex tasks that require more thanone action, where sequential decision-making (SDM) capa-bilities become necessary. The key contribution of this workis a robot SDM framework, called LCORPP, that supportsthe simultaneous capabilities of supervised learning for pas-sive state estimation, automated reasoning with declarativehuman knowledge, and planning under uncertainty towardachieving long-term goals. In particular, we use a hybrid rea-soning paradigm to refine the state estimator, and provideinformative priors for the probabilistic planner. In experi-ments, a mobile robot is tasked with estimating human in-tentions using their motion trajectories, declarative contex-tual knowledge, and human-robot interaction (dialog-basedand motion-based). Results suggest that, in efficiency and ac-curacy, our framework performs better than its no-learningand no-reasoning counterparts in office environment.

1 IntroductionMobile robots have been able to operate in everyday en-vironments over extended periods of time, and travel longdistances that have been impossible before, while providingservices, such as escorting, guidance, and delivery (Hawes etal. 2017; Veloso 2018; Khandelwal et al. 2017). Sequentialdecision-making (SDM) plays a key role toward robot long-term autonomy, because real-world domains are stochastic,and a robot must repeatedly estimate the current world stateand decide what to do next.

There are at least three AI paradigms, namely super-vised learning, automated reasoning, and probabilistic plan-ning, that can be used for robot SDM. Each of the threeparadigms has a long history with rich literature. However,none of the three completely meet the requirements in thecontext of robot SDM. First, a robot can use supervisedlearning to make decisions, e.g., to learn from the demon-strations of people or other agents (Argall et al. 2009).However, the methods are not designed for reasoning withdeclarative contextual knowledge that are widely availablein practice. Second, knowledge representation and reason-ing (KRR) methods can be used for decision making (Gel-fond and Kahl 2014). However, such knowledge can hardly

Copyright © 2020, Association for the Advancement of ArtificialIntelligence (www.aaai.org). All rights reserved.

be comprehensive in practice, and robots frequently find itdifficult to survive from inaccurate or outdated knowledge.Third, probabilistic planning methods support active infor-mation collection for goal achievement, e.g., using decision-theoretic frameworks such as Markov decision processes(MDPs) (Puterman 2014) and partially observable MDPs(POMDPs) (Kaelbling, Littman, and Cassandra 1998). How-ever, the planning frameworks are ill-equipped for incorpo-rating declarative contextual knowledge.

In this work, we develop a robot SDM framework that en-ables the simultaneous capabilities of learning from past ex-periences, reasoning with declarative contextual knowledge,and planning toward achieving long-term goals. Specifically,we use long short-term memory (LSTM) (Hochreiter andSchmidhuber 1997) to learn a classifier for passive stateestimation using streaming sensor data, and use common-sense reasoning and probabilistic planning (CORPP) (Zhangand Stone 2015) for active perception and task completionsusing contextual knowledge and human-robot interaction.Moreover, the dataset needed for supervised learning canbe augmented through the experience of active human-robotcommunication, which identifies the second contribution ofthis work. The resulting algorithm is called learning-CORPP(LCORPP), as overviewed in Figure 1.

We apply LCORPP to the problem of human intention es-timation using a mobile robot. The robot can passively es-timate human intentions based on their motion trajectories,reason with contextual knowledge to improve the estima-tion, and actively confirm the estimation through human-robot interaction (dialog-based and motion-based). Resultssuggest that, in comparison to competitive baselines fromthe literature, LCORPP significantly improves a robot’s per-formance in state estimation (in our case, human intention)in both accuracy and efficiency. 1

2 Related WorkKRR and SDM: SDM algorithms can be grouped into twoclasses depending on the availability of the world model,namely probabilistic planning, and reinforcement learning(RL). Next, we select a sample of the SDM algorithms, and

1A demo video is available: https://youtu.be/YgiB1fpJgmo

arX

iv:1

901.

0532

2v3

[cs

.AI]

10

Dec

201

9

Reporting action

Classifier

Probabilistic rules

Contextual knowledge

Domain expert

Observation (passive)

Belief

Data

Policy

World

Act

ion

Obs

erva

tion

(act

ive)

Training

Data augmentation

Supervised Learning Automated Reasoning Probabilistic Planning

- Streaming data is from world as well- Belief should be inside planner- Passive vs. active interaction- Remove redundant info- Text on one side of the lines- Vertical text heading to the left

Streaming sensor data

Cross validation

Figure 1: An overview of LCORPP that integrates supervised learning, automated reasoning, and probabilistic planning. Streaming sensordata, e.g., from RGB-D sensors, is fed into an offline-trained classifier. The classifier’s output is provided to the reasoner along with classifier’scross-validation. This allows the reasoner to encode uncertainty of the classifier’s output. The reasoner reasons with declarative contextualknowledge provided by the domain expert, along with the classifier’s output and accuracies in the form of probabilistic rules, and produces aprior belief distribution for the probabilistic planner. The planner suggests actions to enable the robot to actively interact with the world, anddetermines what actions to take, including when to terminate the interaction and what to report. At each trial, the robot collects the informationgathered from the interaction with the world. In case of limited experience for training, LCORPP supports data augmentation through activelyseeking human supervision. Solid and dashed lines correspond to Algorithms 1 and 2 respectively, as detailed in Section 4.

make comparisons with LCORPP.When world model is unavailable, SDM can be realized

using RL. People have developed algorithms for integrat-ing KRR and RL methods (Leonetti, Iocchi, and Stone 2016;Yang et al. 2018; Lyu et al. 2019; Sridharan and Rainge2014; Griffith et al. 2013; Illanes et al. 2019). Among them,Leonetti, Iocchi, and Stone used declarative action knowl-edge to help an agent to select only the reasonable actions inRL exploration. Yang et al. developed an algorithm calledPEORL that integrates hierarchical RL with task planning,and this idea was applied to deep RL by Lyu et al. in 2019.

When world models are provided beforehand, one canuse probabilistic planning methods for SDM. Contextualknowledge and declarative reasoning have been used tohelp estimate the current world state in probabilistic plan-ning (Zhang, Sridharan, and Wyatt 2015; Zhang and Stone2015; Sridharan et al. 2019; Ferreira et al. 2017; Chitnis,Kaelbling, and Lozano-Perez 2018; Lu et al. 2018; Lu et al.2017). Work closest to this research is the CORPP algorithm,where hybrid reasoning (both logical and probabilistic) wasused to guide probabilistic planning by calculating a proba-bility for each possible state (Zhang and Stone 2015). Morerecently, researchers have used human-provided declarativeinformation to improve robot probabilistic planning (Chit-nis, Kaelbling, and Lozano-Perez 2018).

The main difference from the algorithms is that LCORPPis able to leverage the extensive data of (labeled) decision-making experiences for continuous passive state estimation.In addition, LCORPP supports collecting more data from thehuman-robot interaction experiences to further improve thepassive state estimator over time.KRR and supervised learning: Reasoning methods havebeen incorporated into supervised learning in natural lan-guage processing (NLP) (Chen et al. 2019; Zellers, Holtz-man, and others 2019) and computer vision (Zellers et al.2019; Aditya et al. 2019; Chen et al. 2018) among others.For instance, Chen et al. in 2019 used commonsense knowl-edge to add missing information in incomplete sentences(e.g., to expand “pour me water” by adding “into a cup” tothe end); and Aditya et al. used spatial commonsense in the

Viual Question Answering (VQA) tasks. Although LCORPPincludes components for both KRR and supervised learn-ing, we aim at a SDM framework for robot decision-makingtoward achieving long-term goals, which identifies the keydifference between LCORPP and the above-mentioned algo-rithms.SDM and supervised learning: Researchers have devel-oped various algorithms for incorporating human super-vision into SDM tasks (Taylor and Borealis 2018; Amiriet al. 2018; Thomaz, Breazeal, and others 2006). Amongthem, Amiri et al. used probabilistic planning for SDM un-der mixed observability, where the observation model waslearned from annotated datasets. Taylor and Borealis sur-veyed a few ways of improving robots’ SDM capabilitieswith supervision (in the form of demonstration or feedback)from people. In comparison to the above methods, LCORPPis able to leverage human knowledge, frequently in declara-tive forms, toward more efficient and accurate SDM.

To the best of our knowledge, this is the first work onrobot SDM that simultaneously supports supervised learningfor passive perception, automated reasoning with contextualknowledge, and active information gathering toward achiev-ing long-term goals under uncertainty.

3 BackgroundIn this section, we briefly summarize the three computa-tional paradigms used in the buildings blocks of LCORPP.

LSTM: Recurrent neural networks (RNNs) (Hopfield 1982)are a kind of neural networks that use their internal state(memory) to process sequences of inputs. Given the inputsequence vector (x0, x1, ..., xn), at each time-step, a hid-den state is produced that results in the hidden sequence of(h0, h1, ..., hn). LSTM (Hochreiter and Schmidhuber 1997)network, is a type of RNN that includes LSTM units. Eachmemory unit in the LSTM hidden layer has three gates formaintaining the unit state: input gate defines what informa-tion is added to the memory unit; output gate specifies whatinformation is used as output; and forget gate defines whatinformation can be removed. LSTMs use memory cells to

resolve the problem of vanishing gradients, and are widelyused in problems that require the use of long-term con-textual information, e.g., speech recognition (Graves, Mo-hamed, and Hinton 2013) and caption generation (Vinyals etal. 2015). We use LSTM-based supervised learning for pas-sive state estimation with streaming sensor data in this work.

P-log: Answer Set Prolog (ASP) is a logic programmingparadigm that is strong in non-monotonic reasoning (Gel-fond and Kahl 2014; Lifschitz 2016). An ASP program in-cludes a set of rules, each in the form of:

l0 ← l1, · · · , ln, not lk, · · · , not ln+k

where l’s are literals that represent whether a statement istrue or not, and symbol not is called default negation. Theright side of a rule is the body, and the left side is the head.A rule head is true if the body is true.

P-log extends ASP by allowing probabilistic rules forquantitative reasoning. A P-log program consists of the log-ical and probabilistic parts. The logical part inherits the syn-tax and semantics of ASP. The probabilistic part containspr-atoms in the form of:

prr(G(η) = w|B) = v

where G(η) is a random variable, B is a set of literals andv ∈ [0, 1]. The pr-atom states that, ifB holds and experimentr is fixed, the probability of G(η) = w is v.

POMDPs: Markov decision processes (MDPs) can be usedfor sequential decision-making under full observability. Par-tially observable MDPs (POMDPs) (Kaelbling, Littman, andCassandra 1998) generalize MDPs by assuming partial ob-servability of the current state. A POMDP model is repre-sented as a tuple (S,A, T,R, Z,O, γ) where S is the state-space, A is the action set, T is the state-transition function,R is the reward function, O is the observation function, Z isthe observation set and γ is discount factor that determinesthe planning horizon.

An agent maintains a belief state distribution b with ob-servations (z ∈ Z) using the Bayes update rule:

b′(s′) =O(s′, a, z)

∑s∈S T (s, a, s′)b(s)

pr(z|a, b)where s is the state, a is the action, pr(z|a, b) is a normalizer,and z is an observation. Solving a POMDP produces a policythat maps the current belief state distribution to an actiontoward maximizing long-term utilities.

CORPP (Zhang and Stone 2015) uses P-log (Baral, Gel-fond, and Rushton 2009) for knowledge representation andreasoning, and POMDPs (Kaelbling, Littman, and Cassan-dra 1998) for probabilistic planning. Reasoning with a P-logprogram produces a set of possible worlds, and a distributionover the possible worlds. In line with the CORPP work, weuse P-log for reasoning purposes, while the reasoning com-ponent is not restricted to any specific declarative language.

4 FrameworkIn this section, we first introduce a few definitions, thenfocus on LCORPP, our robot SDM framework, and finallypresent a complete instantiation of LCORPP in detail.

Algorithm 1 LCORPP

Require: A set of instance-label pairs Ω (training dataset),Logical-probabilistic rules θ, Termination probability thresh-old ε, POMDP modelM, Batch size K

1: Compute classifier ρ using Ω2: C ← CROSS-VALIDATE(ρ), where C is a confusion matrix3: Update rules in θ with the probabilities in C4: Compute action policy π withM (using POMDP solvers)5: Initialize C (same shape of C) using uniform distributions6: Initialize counter: c← 07: while SSUB(C − C) > ε do8: Collect new instance I9: L← DT-CONTROL(ρ, I,M), where L is the label of I

10: Store instance-label pair: ω ← (I, L)11: Ω← Ω ∪ ω12: c← c+ 113: if c == K then14: Compute classifier ρ using augmented dataset Ω

15: C ← CROSS-VALIDATE(ρ)

16: Update rules in θ with the probabilities in C17: C ← C18: c← 019: end if20: end while

Definitions: Before describing the algorithm, it is neces-sary to define three variable sets of Vlrn, Vrsn and Vpln thatare modeled in the learning, reasoning, and planning com-ponents respectively. For instance, Vlrn includes a finite setof variables:

Vlrn = V lrn0 , V lrn

1 , ...We consider factored spaces, so the three variable sets can

be used to specify the three state spaces respectively, i.e.,Slrn, Srsn and Spln. For instance, the learning component’sstate space, Slrn, can be specified by Vlrn, and includes afinite set of states in the form of:

Slrn = slrn0 , slrn1 , ...

Building on the above definitions, we next introduce theLCORPP algorithm followed by a complete instantiation.

4.1 AlgorithmsWe present LCORPP in the form of Algorithms 1 and 2,where Algorithm 1 calls the other.

The input of LCORPP includes the dataset Ω for sequenceclassification, declarative rules θ, logical facts β, and aPOMDP model M. Each sample in Ω is a matrix of sizeT × N , where T is the time length and N is the number offeatures. Each sample is associated with a label, where eachlabel corresponds to state slrn ∈ Slrn. Logical-probabilisticrules, θ, are used to represent contextual knowledge fromhuman experts. Facts, β, are collected at runtime, e.g., cur-rent time and location, and are used together with the rulesfor reasoning. Finally, POMDP modelM includes world dy-namics and is used for planning under uncertainty towardactive information gathering and goal accomplishments.

Algorithm 1 starts with training classifier ρ using datasetΩ, and confusion matrix C that is generated by cross-validation. The probabilities in C are passed to the reasoner

Algorithm 2 DT-CONTROL: Decision Theoretic ControlRequire: Classifier ρ, Instance I , and POMDP modelM1: Update state: slrn ← ρ(I), where slrn ∈ Slrn

2: Collect facts β from the world, and add slrn into β3: V

rsn ← V |V ∈ Vrsn, and V ∈ Vpln4: Use V

rsnto form the state space of Srsn, where Srsn ⊂ Srsn

and Srsn ⊂ Spln

5: Reason with θ and β to compute belief brsn over Srsn

6: Compute brsn (over Srsn), i.e., a marginal of brsn

7: Initialize belief bpln (over Spln) using brsn

8: repeat9: Select action a← π(bpln), and execute a

10: Make observation o11: Update bpln based on a and o using Bayes update rule12: until reaching terminal state term ∈ Spln

13: L← EXTRACTLABEL(a)

to update probabilistic rules θ. Action policy π is then gen-erated using the POMDP modelM and off-the-shelf solversfrom the literature. Matrix C (of shape C) and counter care initialized with uniform distribution and 0 respectively.SSUB(C − C) is a function that sums up the absolute val-ues of the element-wise subtraction of matrices C and C.As long as the output of SSUB is greater than the termina-tion threshold ε, the robot collects a new instance I , passesthe instance along with the classifier and the model to Al-gorithm 2 to get label L (Lines 8-9). The pair of instanceand label, ω, is added to the dataset Ω and the counter (c)is incremented. If c reaches the batch size K, new classifierρ is trained using the augmented dataset. Then, it is cross-validated to generate C, and the rules θ are updated. Also, cis set to 0 for collection of a new batch of data (Lines 13-18).

Algorithm 2 requires classifier ρ, data instance I , andPOMDP model M . The classifier (ρ) outputs the learner’scurrent state, slrn ∈ Slrn. This state is then merged intoβ in Line 2, which is later used for reasoning purposes. Aset of variables, V

rsn, is constructed in Line 3 to form state

space Srsn, which is a partial state space of both Srsn andSpln. brsn is the reasoner’s posterior belief. Belief distri-bution brsn over Srsn bridges the gap between LCORPP’sreasoning and planning components: brsn is computed asa marginal distribution of the reasoner’s output in Line 6;and used for generating the prior distribution of bpln for ac-tive interactions. LCORPP initializes POMDP prior belief bpln

over the state set Spln with brsn in Line 7, and uses policyπ to map bpln to actions. This sense-plan-act loop contin-ues until reaching the terminal state and the estimation labelwould be extracted from the reporting action (described indetail in subsection 4.2).

4.2 InstantiationWe apply our general-purpose framework to the problem ofhuman intention estimation using a mobile robot, as shownin Figure 2. The robot can observe human motion trajecto-ries using streaming sensor data (RGB-D images), has con-textual knowledge (e.g., visitors tend to need guidance help),and is equipped with dialog-based and motion-based inter-

May I help you?

3rd person view robot’s view

Figure 2: Robot estimating human intention, e.g., human intend-ing to interact or not, by analyzing human trajectories, reasoningwith contextual knowledge (such as location and time), and takinghuman-robot interaction actions.

action capabilities. The objective is to determine human in-tention (in our case, whether a human is interested in in-teraction or not). In the following subsections, we providetechnical details of a complete LCORPP instantiation in thisdomain.

Learning for Perception with Streaming Data In orderto make correct state estimation based on the streaming sen-sor data while considering the dependencies at various timesteps, we first train and evaluate the classifier ρ using datasetΩ. We split the dataset into training and test sets, and pro-duce the confusion matrix C, which is later needed by thereasoner. Human intention estimation is modeled as a clas-sification problem for the LSTM-based learner:

slrn = ρ(I)

where robot is aiming at estimating slrn ∈ Slrn usingstreaming sensor data I . In our case, streaming data is inthe form of people motion trajectories; and there exists onlyone binary variable, intention ∈ Vlrn. As a result, state setSlrn includes only two states:

Slrn = slrn0 , slrn1

where slrn0 and slrn1 correspond to the person having in-tention of interacting with the robot or not. Since the hu-man trajectories are in the form of sequence data, we useLSTM to train a classifier for estimating human intentionswith motion trajectories. Details of the classifier training arepresented in Section 5. Next, we explain how the classifieroutput is used for reasoning.

Reasoning with Contextual Knowledge Contextualknowledge, provided by domain experts, can help the robotmake better estimations. For instance, in the early morningsof work days, people are less likely to be interested ininteracting with the robot, in comparison to the universityopen-house days. The main purpose of the reasoningcomponent is to incorporate such contextual knowledge tohelp the passive state estimation.

The knowledge base consists of logical-probabilistic rulesθ in P-log (Baral, Gelfond, and Rushton 2009), a declarativelanguage that supports the representation of (and reasoningwith) both logical and probabilistic knowledge. The reason-ing program consists of random variables set Vrsn. It startscollecting facts and generating β. The confusion matrix gen-erated by the classifier’s cross-validation is used to update

θ. The variables that are shared between the reasoning andplanning components are in the set V

rsn. The reasoner pro-

duces a belief brsn over Srsn via reasoning with θ and β.In the problem of human intention estimation, the rea-

soner contains random variables:Vrsn = location, time, identity, intention, · · · ,where the range of each variable is defined as below:

location: classroom, librarytime: morning, afternoon, eveningidentity: student, professor, visitorintention: interested, not interested

We further include probabilistic rules into the reasoningcomponent. For instance, the following two rules state thatthe probability of a visitor showing up in the afternoon is 0.7,and the probability of a professor showing up in the library(instead of other places) is 0.1, respectively.pr(time=afternoon|identity=visitor)=0.7.

pr(location=library|identity=professor)=0.1.

It should be noted that time and location are facts thatare fully observable to the robot, whereas human identityis a latent variable that must be inferred. Time, location, andintention are probabilistically determined by human identity.We use time and location to infer human identity, and thenestimate human intention.

The binary distribution over human intention, brsn, amarginal distribution of brsn over Srsn, is provided to thePOMDP-based planner as informative priors.2

Active Perception via POMDP-based HRI Robots canactively take actions to reach out to people and gatherinformation. We use POMDPs to build probabilistic con-trollers. A POMDP model can be represented as a tuple(Spln, A, T,R, Z,O, γ). We briefly discuss how each com-ponent is used in our models:

• Spln : Srsn×Splnl ∪term is the state set. Srsn includes

two states representing human being interested to interactor not. Spln

l includes two states representing whether therobot has turned towards the human or not and term isthe terminal state.

• A : Aa∪Ar is the set of actions.Aa includes both motion-based and language-based interaction actions, includingturning (towards the human), greeting, and moving for-ward slightly. Ar includes two actions for reporting thehuman being interested in interaction or not.

• T (s, a, s′) = P (s′|s, a) is the transition function that ac-counts for uncertain or non-deterministic action outcomeswhere a ∈ A and s ∈ S. Reporting actions deterministi-cally lead to the term state.2The reasoning component can be constructed using other

logical-probabilistic paradigms that build on first-order logic, suchas Probabilistic Soft Logic (PSL) (Bach et al. 2017) and MarkovLogic Network (MLN) (Richardson and Domingos 2006). In com-parison, P-log directly takes probabilistic, declarative knowledgeas the input, instead of learning weights with data, and meets ourneed of utilizing human knowledge in declarative forms.

Figure 3: Tracking area and sensor setup in a shopping mall .Thedashed line shows the area covered by the sensors. The photos onright show the corridor area in the afternoon on a weekday (bottom-left) and weekend (bottom-right) [Kato et al., 2015].

• Z = pos, neg, none is the observation set modelinghuman feedback in human-robot interaction.

• O(s′, a, z) = P (z|a, s′), where z ∈ Z, is the observa-tion function, which is used for modeling people’s noisyfeedback to the robot’s interaction actions.

• R(s, a) is the reward function, where costs of interactionactions, aa ∈ Aa, correspond to the completion time. Acorrect (wrong) estimation yields a big bonus (penalty).

Reporting actions deterministically lead to the term state.We use a discount factor γ = 0.99 to give the robot a longplanning horizon. Using an off-the-shelf solver (e.g., (Kur-niawati, Hsu, and Lee 2008)), the robot can generate a be-havioral policy that maps its belief state to an action towardefficiently and accurately estimating human intentions.

To summarize, the robot’s LSTM-based classifier esti-mates human intention based on the human trajectories.The reasoner uses human knowledge to compute a distribu-tion on human intention. The reasoner’s intention estimationserves as the prior of the POMDP-based planner, which en-ables the robot to actively interact with people to figure outtheir intention. The reasoning and planning components ofCORPP are constructed using human knowledge, and do notinvolve learning. The reasoning component aims at correct-ing and refining the LSTM-based classifier’s output, and theplanning component is for active perception.

5 ExperimentsIn this section, we describe the testing domain (including thedataset), experiment setup, and statistical results.

5.1 Dataset and Learning ClassifiersFigure 3 shows the shopping center environment, where thehuman motion dataset (Kato, Kanda, and Ishiguro 2015) wascollected using multiple 3D range sensors mounted over-head. We use the dataset to train the LSTM-based classi-fier. Each instance in the dataset includes a human motiontrajectory in 2D space, and a label of whether the humaneventually interacts with the robot or not. There are totally2286 instances in the dataset, where 63 are positive instances

Figure 4: (Left) A human detected and tracked by our Segway-based robot in the classroom building. (Right) The correspondingcollected trajectory, where the robot’s position is shown in square“”, and the human trajectory starts at dot and finishes in “4”position. In this example, the human was interested in interactions.

(2.7%). Each trajectory includes a sequence of data fieldswith the sampling rate of 33 milliseconds. Each data field isin the form of a vector: (xi, yi, zi, vi, θmi

, θhi). Index i

denotes the timestep. xi and yi are the coordinates in mil-limeter. zi is the human height. vi is human linear velocityin mm/s. θmi is the motion angle in radius. θhi is the faceorientation in radius. We only use the x and y coordinates,because of the limitations of our robot’s perception capabil-ities.

The input vector length is 60 including 30 pairs of xand y values. Our LSTM’s hidden layer includes 50 mem-ory units. In order to output binary classification results, weuse a dense layer with sigmoid activation function in theoutput layer. We use Adam (Kingma and Ba 2014), a first-order gradient method, for optimization. The loss functionwas calculated using binary cross-entropy. For regulariza-tion, we use a dropout value of 0.2. The memory units andthe hidden states of the LSTM are initialized to zero. Theepoch size (number of passes over the entire training data)is 300. The batch size is 32. The data was split into sets fortraining (70%) and testing (30%). To implement the classi-fier training, we used Keras (Chollet and others 2015), anopen-source python deep-learning library.

5.2 Illustrative ExampleConsider an illustrative example: a visitor to a classroombuilding in the afternoon was interested to interact with therobot. The robot’s goal is to identify the person’s intentionas efficiently and accurately as possible.

Human motion trajectory is captured by our robot usingRGB-D sensors. Figure 4 presents a detected person, andthe motion trajectory. The trajectory is passed to the LSTM-based classifier, which outputs the person being not inter-ested in interaction (false negative).

The robot then collected facts about time and location.Domain knowledge enables the robot to be aware that: pro-fessors and students are more likely to show up in the class-room; and visitors are more likely to show up in the after-noon and to interact with the robot, whereas they are lesslikely to be present in the classroom building. Also, theLSTM classifier’s confusion matrix, as shown in Figure 5,is encoded as a set of probabilistic rules in P-log, wherethe true-negative probability is 0.71. Therefore, given all thedeclarative contextual knowledge, the reasoner computes thefollowing distribution over the variable of human identity,

V rsnid , in the range of [student, visitor, professor]:

Dist(V rsnid ) = [0.36, 0.28, 0.36] (1)

True FalsePrediction

Tru

eFals

eG

round T

ruth 0.76 0.24

0.29 0.71

Figure 5: Confusion ma-trix of the LSTM classifier

Given the observed facts andthe classifier’s output, the robotqueries its reasoner to estimatethe distribution over possible hu-man intentions

brsn == [0.22, 0.78] (2)

where 0.22 corresponds to the hu-man being interested in interac-tion. brsn is the belief over stateset Srsn (a marginal distribution of both Srsn and Spln).

The reasoner’s output of brsn is used for initializing thebelief distribution, bpln, for the POMDP-based planner:

bpln = [0.22, 0.78, 0, 0, 0]

where bpln is over the state set of Spln as described in Sec-tion 4.2. For instance, spln0 ∈ Spln is the state where therobot has not taken the “turn” action, and the human is inter-ested in interaction. Similarly, spln3 ∈ Spln is the state wherethe robot has taken the action “turn”, and the human is notinterested in interaction.

Figure 6: An illustrative example of information flow in LCORPP:the learning component generates “facts” and probabilistic rules forthe reasoner, and the reasoner computes a prior distribution for theplanner, which actively interacts with the environment.

During the action-observation cycles (in simulation), pol-icy π maps bpln to “greet” and “move forward” actions, andbpln is updated until the robot correctly reported human in-tention and reached terminal state (spln4 ). The actions, corre-sponding human feedback, and belief update are presentedin Figure 6. Although, the LSTM classifier made a wrongestimation, the reasoner and planner helped the robot suc-cessfully recover from the wrong estimation.

5.3 Experimental ResultsWe did pairwise comparisons between LCORPP with the fol-lowing methods for human intention estimation to investi-gate several hypotheses. Our baselines include: Learning(L): learned classifier only. Reasoning (R): reasoning with

L R L+R P R+P(Ours)LCORPP

Strategy

0.5

0.6

0.7

0.8

0.9

1.0F1

Sco

re

P R+P (Ours)LCORPP

Strategy

6

7

8

9

10

11

12

13

Cost

Figure 7: Pairwise comparisons of LCORPP with five baseline se-quential decision-making strategies. The right subfigure excludesthe strategies that do not support active human-robot interactionand hence produce zero costs.

contextual knowledge. Planning (P): POMDP-based interac-tion with uniform priors. Learning+Reasoning (L+R): rea-soning with the classifier’s output and knowledge. Reason-ing+Planning (R+P): reasoning with knowledge and plan-ning with POMDPs.

Our experiments are designed to evaluate the followinghypotheses. I) Given that LCORPP’s prior belief is generatedusing reasoner and learner, it would outperform baselines inintention estimation in F1 score while receiving less actioncosts; II) In case of inaccurate knowledge, or III) the learnernot having a complete sensor input, LCORPP’s planner cancompensate these imperfections by taking more actions tomaintain higher F1 score. IV) In case of scarcity of data,robot’s most recent interaction can be used to augment thedataset and improve overall performance of LCORPP.

In each simulated trial, we first sample human iden-tity randomly, and then sample time and location accord-ingly, using contextual knowledge, such as professors tendto showing up early. According to the time, location, andidentity, we sample human intention. Finally, we sample atrajectory from the test set of the dataset, according to thepreviously sampled human intention. We added 30% noiseto the human reactions (robot’s observation) being compli-ant with the ground truth but independent from robot’s ac-tions. LCORPP requires considerable computation time fortraining the classifier (∼10 min) and generating the POMDP-based action policy (∼1 min). The training and policy gen-eration are conducted offline, so they do not affect the run-time efficiency. Reasoning occurs at runtime, and typicallyrequires less than 1 millisecond.

LCORPP vs. Five Baselines: Figure 7 shows the overallcomparisons using the six SDM strategies, each number isan average of 5000 trials, where the setting is the samein all following experiments. The three strategies, P, R+Pand LCORPP, include the POMDP-based planning compo-nent, and perform better than no-planning baselines in F1score. Among the planning strategies, ours produces the bestoverall performance in F1 score, while reducing the interac-tion costs (dialog-based and motion-based)(Hypothesis I).

Inaccurate Knowledge: In this experiment, we evaluatethe robustness of LCORPP to inaccurate knowledge (Hy-pothesis II). Our hypothesis is that, in case of contextual

Low Medium HighKnowledge level

0.3

0.4

0.5

0.6

0.7

0.8

0.9

F1 S

core

Low Medium HighKnowledge level

7.5

8.0

8.5

9.0

9.5

10.0

10.5

Cost

R L+R R+P LCORPP (Ours)

Figure 8: Performances of LCORPP and baselines given contextualknowledge of different accuracy levels: High, Medium and Low.Baselines that do not support reasoning with human knowledge arenot included in this experiment.

knowledge being inaccurate, LCORPP is capable of recover-ing via actively interacting with people. We used knowledgebases (KBs) of different accuracy levels: High, Medium,and Low. A high-accuracy KB corresponds to the groundtruth. Medium- and low-accuracy KBs are incomplete, andmisleading respectively. For instance, low-accuracy knowl-edge suggests that professors are more likely to interact withthe robot, whereas visitors are not, which is opposite to theground truth.

Figure 8 shows the results where we see the performancesof R and L+R baseline strategies drop to lower than 0.4 inF1 score, when the contextual knowledge is of low accuracy.In F1 score, neither R+P nor LCORPP is sensitive to low-accuracy knowledge, while LCORPP performs consistentlybetter than R+P. In particular, when the knowledge is of lowaccuracy, LCORPP retains the high F1 score (whereas R+Pcould not) due to its learning component.

Additional experimental results on Hypotheses III and IVare provided in the supplementary document.

6 Conclusions & Future Work

In this work, we develop a robot sequential decision-making framework that integrates supervised learning forpassive state estimation, automated reasoning for incorpo-rating declarative contextual knowledge, and probabilisticplanning for active perception and task completions. Thedeveloped framework has been applied to a human inten-tion estimation problem using a mobile robot. Results sug-gest that the integration of supervised deep learning, logical-probabilistic reasoning, and probabilistic planning enablessimultaneous passive and active state estimation, producingthe best performance in estimating human intentions.

This work enables research directions for future work. Itcan be interesting to consider situations where contradic-tions exist among LCORPP’s components, e.g., the classi-fier’s output consistently does not agree with the reasoningresults from human knowledge. Additionally, we can lever-age multi-modal perception for cases where people approachthe robot outside its camera sensor range. When the robothas to estimate the intentions of multiple humans, more ad-vanced human-robot interaction methods can be deployed.

AcknowledgmentsThis work has taken place in the Autonomous Intelligent Robotics(AIR) Group at SUNY Binghamton. AIR research is supportedin part by grants from the National Science Foundation (IIS-1925044), Ford Motor Company, and SUNY Research Foundation.

References[Aditya et al. 2019] Aditya, S.; Saha, R.; Yang, Y.; and Baral, C.

2019. Spatial knowledge distillation to aid visual reasoning. InWACV, 227–235.

[Amiri et al. 2018] Amiri, S.; Wei, S.; Zhang, S.; Sinapov, J.;Thomason, J.; and Stone, P. 2018. Multi-modal predicate iden-tification using dynamically learned robot controllers. In IJCAI,4638–4645.

[Argall et al. 2009] Argall, B. D.; Chernova, S.; Veloso, M.; andBrowning, B. 2009. A survey of robot learning from demonstra-tion. Robotics and autonomous systems 57(5):469–483.

[Bach et al. 2017] Bach, S. H.; Broecheler, M.; Huang, B.; andGetoor, L. 2017. Hinge-loss markov random fields and prob-abilistic soft logic. The Journal of Machine Learning Research18(1):3846–3912.

[Baral, Gelfond, and Rushton 2009] Baral, C.; Gelfond, M.; andRushton, N. 2009. Probabilistic reasoning with answer sets. TPLP9(1):57–144.

[Chen et al. 2018] Chen, X.; Li, L.-J.; Fei-Fei, L.; and Gupta, A.2018. Iterative visual reasoning beyond convolutions. In CVPR.

[Chen et al. 2019] Chen, H.; Tan, H.; Kuntz, A.; Bansal, M.; andAlterovitz, R. 2019. Enabling robots to understand incompletenatural language instructions using commonsense reasoning. arXivpreprint arXiv:1904.12907.

[Chitnis, Kaelbling, and Lozano-Perez 2018] Chitnis, R.; Kael-bling, L. P.; and Lozano-Perez, T. 2018. Integrating human-provided information into belief state representation usingdynamic factorization. arXiv preprint arXiv:1803.00119.

[Chollet and others 2015] Chollet, F., et al. 2015. Keras. https://keras.io.

[Ferreira et al. 2017] Ferreira, L. A.; Bianchi, R. A.; Santos, P. E.;and de Mantaras, R. L. 2017. Answer set programming fornon-stationary markov decision processes. Applied Intelligence47(4):993–1007.

[Gelfond and Kahl 2014] Gelfond, M., and Kahl, Y. 2014. Knowl-edge representation, reasoning, and the design of intelligentagents: The answer-set programming approach. Cambridge Uni-versity Press.

[Graves, Mohamed, and Hinton 2013] Graves, A.; Mohamed, A.-r.;and Hinton, G. 2013. Speech recognition with deep recurrentneural networks. In IEEE International Conference on Acoustics,Speech and Signal Processing.

[Griffith et al. 2013] Griffith, S.; Subramanian, K.; Scholz, J.; Is-bell, C. L.; and Thomaz, A. L. 2013. Policy shaping: Integratinghuman feedback with reinforcement learning. In NIPS.

[Hawes et al. 2017] Hawes, N.; Burbridge, C.; Jovan, F.; et al. 2017.The strands project: Long-term autonomy in everyday environ-ments. IEEE Robotics & Automation Magazine 24(3):146–156.

[Hochreiter and Schmidhuber 1997] Hochreiter, S., and Schmidhu-ber, J. 1997. Long short-term memory. Neural computation9(8):1735–1780.

[Hopfield 1982] Hopfield, J. J. 1982. Neural networks and physicalsystems with emergent collective computational abilities. Proceed-ings of the national academy of sciences 79(8):2554–2558.

[Huggins-Daines et al. 2006] Huggins-Daines, D.; Kumar, M.;Chan, A.; Black, A. W.; Ravishankar, M.; and Rudnicky, A. I. 2006.Pocketsphinx: A free, real-time continuous speech recognition sys-tem for hand-held devices. In 2006 IEEE International Conferenceon Acoustics Speech and Signal Processing Proceedings, volume 1,I–I. IEEE.

[Illanes et al. 2019] Illanes, L.; Yan, X.; Icarte, R. T.; and McIlraith,S. A. 2019. Symbolic planning and model-free reinforcementlearning: Training taskable agents. In 4th Multidisciplinary Con-ference on RLDM.

[Kaelbling, Littman, and Cassandra 1998] Kaelbling, L. P.;Littman, M. L.; and Cassandra, A. R. 1998. Planning and actingin partially observable stochastic domains. Artificial intelligence101(1-2):99–134.

[Kato, Kanda, and Ishiguro 2015] Kato, Y.; Kanda, T.; and Ishig-uro, H. 2015. May i help you?: Design of human-like polite ap-proaching behavior. In HRI.

[Khandelwal et al. 2017] Khandelwal, P.; Zhang, S.; Sinapov, J.;Leonetti, M.; Thomason, J.; Yang, F.; et al. 2017. BWIBots: Aplatform for bridging the gap between AI and human–robot inter-action research. The IJRR.

[Kingma and Ba 2014] Kingma, D. P., and Ba, J. 2014. Adam: Amethod for stochastic optimization. arXiv:1412.6980.

[Kurniawati, Hsu, and Lee 2008] Kurniawati, H.; Hsu, D.; and Lee,W. S. 2008. Sarsop: Efficient point-based pomdp planning byapproximating optimally reachable belief spaces. In Robotics: Sci-ence and Systems.

[Leonetti, Iocchi, and Stone 2016] Leonetti, M.; Iocchi, L.; andStone, P. 2016. A synthesis of automated planning and reinforce-ment learning for efficient, robust decision-making. Artificial In-telligence 241:103–130.

[Lifschitz 2016] Lifschitz, V. 2016. Answer sets and the languageof answer set programming. AI Magazine 37(3):7–12.

[Lu et al. 2017] Lu, D.; Zhang, S.; Stone, P.; and Chen, X. 2017.Leveraging commonsense reasoning and multimodal perceptionfor robot spoken dialog systems. In IROS.

[Lu et al. 2018] Lu, K.; Zhang, S.; Stone, P.; and Chen, X. 2018.Robot represention and reasoning with knowledge from reinforce-ment learning. arXiv:1809.11074.

[Lyu et al. 2019] Lyu, D.; Yang, F.; Liu, B.; and Gustafson, S. 2019.Sdrl: interpretable and data-efficient deep reinforcement learningleveraging symbolic planning. In AAAI.

[Puterman 2014] Puterman, M. L. 2014. Markov decision pro-cesses: discrete stochastic dynamic programming. John Wiley &Sons.

[Quigley et al. 2009] Quigley, M.; Conley, K.; Gerkey, B.; Faust, J.;Foote, T.; Leibs, J.; Wheeler, R.; and Ng, A. Y. 2009. Ros: anopen-source robot operating system. In ICRA workshop on opensource software, volume 3, 5. Kobe, Japan.

[Richardson and Domingos 2006] Richardson, M., and Domingos,P. 2006. Markov logic networks. Machine learning 62(1-2):107–136.

[Sridharan and Rainge 2014] Sridharan, M., and Rainge, S. 2014.Integrating reinforcement learning and declarative programming tolearn causal laws in dynamic domains. In International Conferenceon Social Robotics.

[Sridharan et al. 2019] Sridharan, M.; Gelfond, M.; Zhang, S.; andWyatt, J. 2019. Reba: A refinement-based architecture for knowl-edge representation and reasoning in robotics. Journal of ArtificialIntelligence Research.

[Taylor and Borealis 2018] Taylor, M. E., and Borealis, A. 2018.Improving reinforcement learning with human input. In IJCAI,5724–5728.

[Thomaz, Breazeal, and others 2006] Thomaz, A. L.; Breazeal, C.;et al. 2006. Reinforcement learning with human teachers: Evi-dence of feedback and guidance with implications for learning per-formance. In AAAI.

[Veloso 2018] Veloso, M. M. 2018. The increasingly fascinatingopportunity for human-robot-ai interaction: The cobot mobile ser-vice robots. ACM THRI 7(1):5.

[Vinyals et al. 2015] Vinyals, O.; Toshev, A.; Bengio, S.; and Erhan,D. 2015. Show and tell: A neural image caption generator. InCVPR.

[Yang et al. 2018] Yang, F.; Lyu, D.; Liu, B.; and Gustafson, S.2018. Peorl: Integrating symbolic planning and hierarchical re-inforcement learning for robust decision-making. arXiv preprintarXiv:1804.07779.

[Zellers et al. 2019] Zellers, R.; Bisk, Y.; Farhadi, A.; and Choi, Y.2019. From recognition to cognition: Visual commonsense reason-ing. In CVPR.

[Zellers, Holtzman, and others 2019] Zellers, R.; Holtzman, A.;et al. 2019. Hellaswag: Can a machine really finish your sentence?arXiv preprint arXiv:1905.07830.

[Zhang and Stone 2015] Zhang, S., and Stone, P. 2015. Corpp:Commonsense reasoning and probabilistic planning, as applied todialog with a mobile robot. In AAAI, 1394–1400.

[Zhang, Sridharan, and Wyatt 2015] Zhang, S.; Sridharan, M.; andWyatt, J. L. 2015. Mixed logical inference and probabilisticplanning for robots in unreliable worlds. IEEE Transactions onRobotics 31(3):699–713.

Supplementary MaterialsIn this supplementary document, we present additional experimen-tal results, and technical details of LCORPP’s implementation on amobile robot platform.

Partial Motion TrajectoriesWithin the human intention estimation context, the robot’s goal isto identify human intention as accurately and early as possible.However, there is the trade-off between efficiency and accurate. Forinstance, reaching out to people early usually means the robot onlyhas a short piece of human motion trajectory, and frequently pro-duces less accurate intention estimation result. In this experiment,we evaluate how sensitive our platform is to partial human motiontrajectories (Hypothesis III). The experimental results can providerobot practitioners a reference on how early a modern robot canproduce reasonably accurate intention estimation results.

Partial FullTrajectory

0.4

0.5

0.6

0.7

0.8

0.9

1.0

F1 S

core

L L+R LCORPP (Ours)

Figure 9: Experiments with incomplete sensor input (trajectory), incomparison to the two “learning-included” baselines.

We only provided part of each trajectory to the classifier (thefirst quarter in our case). The goal is to evaluate, if the robot isforced to produce the estimation result using only 1/4 piece of thetrajectories, how reliable the three “learning-included” strategiesare, where “L” includes only the LSTM-based classifiers, “L+R”further combines the reasoner, and LCORPP is our approach.

The results are presented in Figure 9. We can see, in case ofpartial trajectories, LCORPP is able to retain its human intentionestimation accuracy in F1 score. We attribute this achievement tothe interaction capability of LCORPP based on experimental results(Figure 8) shown in the main body of the paper.

Active Data AugmentationIn this experiment, we explored the effectiveness of augmenting thedataset (for supervised learning) using the experience of human-robot interaction, including trajectories and corresponding labelsgenerated by the task planner. Our hypothesis is that, as more tra-jectories are collected and added into the dataset, our robot is ableto re-train the classifier, and produce increasingly better overall per-formance in F1 score (Hypothesis IV).

Instead of training the initial classifier using all data, we dividedthe dataset into four batches, each containing 500 instances. In thefirst batch, we ran LCORPP without offering any training data. Asa result, the “supervised learning” component produced a randomclassifier. In the subsequent batches, robot interacts with humanusing the interaction instances collected from early bathes.

Figure 10 shows the results. As the robot becomes more expe-rienced (i.e., more instances of human-robot interaction trials be-come available), LCORPP performs better in terms of F1 score.

1 2 3 4Batch

0.4

0.5

0.6

0.7

0.8

0.9

F1 S

core

LCORPP (No Data) LCORPP (Using Data)

Figure 10: Active data augmentation from the experience ofhuman-robot interaction.

Figure 11: RMP110, the Segway-based mobile robot platform usedfor experimental trials in the real world.

Technical Details in Robot Implementation

We use a Segway-based mobile robot platform, RMP110, for ex-perimental trials in the real world. The platform is equipped withSICK laser sensor for mapping, localization and navigation. As-tra Orbecc RGB-D camera is used for human detection and in-teraction. The robot’s software runs on Robot Operating System(ROS) (Quigley et al. 2009). Our implementation is establishedon the Building Wide Intelligence codebase that is available to thepublic on Github (Khandelwal et al. 2017).

For active information collection, the robot has to take actionswhile receiving (local and unreliable) observations. We manuallyprovide the robot its initial position for localization. After that,to move forward, the robot navigates to a goal position by afixed distance toward to the human. To greet, the robot usesthe “sound play” package in ROS to convert pre-defined text forgreeting into audio. To observe human’s vocal feedback, we usethe pocketsphinx package (Huggins-Daines et al. 2006) forspeech recognition. A set of pre-defined positive utterances indi-cate human’s eagerness to interact.