Error Detection and Surprise in Stochastic Robot Actions

Li Yang Ku, Dirk Ruiken, Erik Learned-Miller and Roderic Grupen

Abstract— For an autonomous robot to accomplish taskswhen the outcome of actions is non-deterministic it is often nec-essary to detect and correct errors. In this work we introduce ageneral framework that stores fine-grained event transitions sothat failures can be detected and handled early in a task. Thesefailures are then recovered through two different approachesbased on whether the error is “surprising” to the robot or not.Surprise transitions are used to create new models that captureobservations previously not in the model. We demonstrate howthe framework is capable of handling uncertainties encounteredby a robot in “pick-and-place” tasks on the uBot-6 mobilemanipulator using both visual and haptic sensor feedback.

I. INTRODUCTION

Building robots that can handle uncertainty is neces-sary for autonomous robots to accomplish tasks in a non-deterministic environment. However, the vast majority offailures either evade detection completely or are detectedonly after a high level action fails to reach the targetstate. This approach makes robots inefficient and can leadto catastrophic failure if the robot continues to execute aplan when the actual state is quite different from what therobot expected. In this work, plans are monitored duringthe execution of low-level actions to create a fine-grainedobserver for error detection. This allows the robot to compareobservations to stored models frequently and to handleunexpected outcomes immediately after they are detected.

In [24], environment, sensors, robots, models, and com-putation are considered as the five factors that give rise touncertainty in robotic applications. Instead of categorizinguncertainty based on the cause, we focus on how it isperceived. Uncertainties a robot may encounter are classifiedinto two categories: 1) random transitions that are withinthe model domain but may still be unlikely given transi-tion probabilities, and, 2) “surprise” transitions that lead tooutcomes that are not represented in the model domain. Anexample of a random transition is the outcome of rolling aloaded die that should come up 6 with high probability, butinstead comes up 1. The robot will not be able to predictthe exact outcome, but by having a model that captures lowprobability outcomes it is capable of handling each of themaccordingly. An example of a surprise transition is if theoutcome of rolling a 6 sided die is 13 while the robot’sdie model has possible outcomes from 1 to 6. In [3], Castidescribes that “surprise can arise only as a consequence ofmodels that are unfaithful to nature.” Therefore we define a“surprise” transition as when the robot observations are not

Li Yang Ku, Dirk Ruiken, Erik Learned-Miller and Roderic Gru-pen are with the College of Information and Computer Sciences,University of Massachusetts Amherst, Amherst, MA 01003, USA{lku,ruiken,elm,grupen}@cs.umass.edu

Fig. 1. The uBot-6 mobile manipulator performing a “what’s up?” gestureto convey that it is surprised after an unexpected event.

explainable by any model in the robot memory. During thesurprise transition a new model of the environment that canexplain this surprising outcome is learned. Failures detectedby the fine-grained observer are then recovered differentlybased on this classification.

In this work, we use the interaction with objects as anexample task with such uncertainties. We model each objectusing a directed multigraph where each node correspondsto a stored observation and each edge corresponds to astored action. We call a stored observation an aspect and thisobject model an aspect transition graph (ATG) model [13].An ATG model captures how actions change observations.We discuss how an ATG model is capable of handlinguncertainties encountered by a robot in “pick-and-place”tasks. This framework is tested on the balancing mobilemanipulator uBot-6 [22]. We demonstrate that the frameworkcan handle random transitions and recover from surprisetransitions at fine-grained time scales.

II. RELATED WORK

In the work done by Rodriguez et al. [21] a classifieris used to predict whether a grasp is a successful graspfor completing a task based on haptic feedback. If it isclassified as a failed grasp the robot aborts and retries.Our approach is based on a similar concept but instead ofrunning a classifier at a specific step we present a generalframework that constantly checks if the observation is withinexpectations. In research done by Donald [4], a theory forerror detection and recovery strategies based on geometryand physical reasoning are introduced. In our work, errordetection and recovery is based on an observation-basedmodel of the environment.

Baldi introduced a computational theory of surprise wheresurprise is defined by the relative entropy between the priorand the posterior distribution of an observer [1]. This formula

is shown to be consistent with what attracts human gaze innatural video stimuli [10]. However, this theory of surprisewould identify informative robot actions that reduces theentropy over models significantly as surprising. In this workwe introduce a simpler definition that agrees better withintuition.

Intrinsic motivators for behavior have long been studiedin psychology. Hull introduced the concept of “drives” suchas hunger, pain, sex, or escape in human behavior, in termsof deficits that the organism wishes to reduce to achievehomeostatic equilibrium [9]. Later, researchers extended theHullian theory by introducing drives for manipulation [7],and for exploration [17]. Berlyne proposed a number of otherintrinsically motivating factors such as novelty, habituation,curiosity, surprise, challenge, and incongruity [2]. In ourwork, we use surprise as an intrinsic motivator to learn newmodels of the environment.

Our work also has many connections to prior work onaffordances defined as “the opportunities for action providedby a particular object or environment.” [5] Affordancesare associative and can be used to infer control actionsfrom observations. Our object models are based on thisinteractionist view of perception and action that focuses onlearning relationships between objects and actions specific tothe robot.

Aspect graphs were first introduced to represent shapein the field of computer vision [12], [6]. An aspect graphcontains distinctive views of an object captured from aviewing sphere centered on the object. The aspect transitiongraph (ATG) introduced in this paper is an extension of thisconcept. In addition to distinctive views, the object modelsummarizes how actions change viewpoints and, thus, thestate of the coupled robot/object system. In addition to visualsensors, extensions to tactile, auditory and other sensorsalso become possible with this representation. The aspecttransition graph model was first introduced by Sen [23]. Inprevious work a mechanism for learning these models byautonomous exploration was introduced using a fixed set ofactions and observations [14], [15]. These models supportbelief-space planning techniques where actions are chosento minimize the expected future model-space entropy andcan be used to condense belief over objects more efficiently.In [13], the aspect transition graph model was extended tohandle an infinite variety of observations and continuousactions.

Our work also relates closely to a body of work on Par-tially Observable Markov Decision Process (POMDP) [11].Since different objects may result in a similar observationfrom certain viewpoints, there is uncertainty in the state ofthe interaction defined by aspects. The goal of a POMDPproblem is to find the optimal policy that maximizes rewardover a finite or infinite horizon. However the computationalcomplexity was shown to be PSPACE-complete over finitehorizons [19] and is undecidable over infinite horizons [16].In this work, the first action on the shortest path to the goalstate is executed based on the most likely current state.

Fig. 2. Part of an aspect transition graph model of a dice. The top rightnode indicates the observation when the robot successfully flipped the dicewhile the bottom right node indicates when the dice slipped. The red circlesindicate the robot hands and the green arrows indicate haptic feedback.

III. MODEL

The finite state controller (FSC) [20] used to solve manyPOMDP problems is defined as a directed graph where eachnode is an action and each edge is an observation. The aspecttransition graph (ATG) model used in our work uses thedual of this graph where each node is an observation andeach edge is an action. An aspect transition graph (ATG)object model is represented using a directed multigraph G =(X ,U), composed of a set of aspect nodes X connected by aset of action edges U that capture the probabilistic transitionbetween aspects. We define an “aspect” as a distinctive setof multiple features called an observation that is stored inthe object model. An action edge U is a triple (X1, X2, A)consisting of a source node X1, a destination node X2 andan action A that transitions between them. Figure 2 showsan example of part of an ATG model that captures possibleobservations when interacting with a die.

Unlike hidden Markov models (HMM) that assume obser-vations are generated from a “true” hidden state such as therobot position, the ATG is an observation-based model thatuses a single observation and the corresponding aspect nodeto represent a robot state directly. The observation-basedapproach avoids modeling and planning with hidden robotstates and uses a model based on observation and memory.More details are described in the Recursive Bayesian Esti-mation subsection.

A. Observation

An ATG model is used to estimate robot state transitionsbased on the robot’s observations; however defining robotstates directly in the observation space of raw sensor outputsis problematic due to the large volume of data in raw sensorfeedback such as images. Therefore, instead of definingobservation based on raw sensor feedback directly, we definean observation as a list of features detected by feature

detectors. Features are patterns in signals that reflect structurein the interaction between the robot and the environment. Weassume that feature detectors are robust and output consistentpatterns under the same conditions. Each feature is presentedwith a binary feature state that indicates whether the featureexists or not and are associated with a Cartesian positionin R3. Position references for multiple features provide anestimate for the object pose with which to scale and orientactions that transition between aspects.

Visual features in the experiment designate: a feature type;the mean Cartesian position for the feature µ in R3; andthe estimated covariance for the feature location Σ in R3×3.For example, the ARtag feature type identifies a unique tagid, the Cartesian coordinate of the center of the tag in therobot (sensor) frame, and the Cartesian covariance of theposition. A red blob feature uniquely identifies the positionand uncertainty of the red robot hand in the visual signal.Haptic feature types include both positions and normals.For positions, µ designates contact positions with covarianceΣ and for normals, µ designates a unit vector in R3 andcovariance Σ in R2×2 represents the uncertainty cone of theunit normal vector. The grasp feature type is computed overa population of contacts. It has two types that designate theforce and moment residuals µ in R3×3 with uncertainty Σ inR3×3. Aspects that define intermediate states in interactionswith an object, for example, are patterns of responses inthese types of random variables that fit the Cartesian templatestored in memory of past interactions with the object. In theexperiments presented in this paper, a simple 28 cm cubeobject is employed with these features distributed over thesix square surfaces of the cube. The size of the set of featurescan be arbitrarily large. For example, the set of featuresthat belongs to the ARtag feature type includes features thatrepresent all possible tag patterns. We describe how we detectfeature types that have a large set of features in the Top DownInference subsection.

An aspect node X in an ATG model represents oneobservation of an object O and stores a list of feature statesdescribed above. We say that two observations are equal if allfeatures that have positive states in one of the observationsalso have positive states in the other observation. There couldbe multiple aspect nodes with the same observation withinone or more ATG models.

B. Action

In an ATG model an action edge U represents an actionthat causes an aspect transition. Actions can be implementedwith open-loop or closed-loop controllers. For each action,its type, parameterization, and reference frame are stored. Arobot has a set of actions available to interact with objects.These actions include visual and haptic servoing actionsas well as gross motor (mobility) actions and fine motor(arm and hand) actions. While mobility actions can be usedto bring an object into reach for manipulation, they alsoalter the view of an object, possibly revealing previouslyhidden visual features. Actions such as pick-up and lift canbe used to manipulate objects, but they can also gather

1: procedure BAYES FILTER(bel(xt−1), at, zt)2: for all xt do3: bel(xt) =

∑xt−1

p(xt|at, xt−1) · bel(xt−1)

4: bel(xt) = p(zt|xt) · bel(xt)5: end for6: NORMALIZE(bel(xt))7: end procedure

Fig. 4. Bayes Filter Algorithm

haptic feedback and change the current viewpoint to revealotherwise unavailable features (e.g. surface markings on thebottom face of a box).

High-level manipulation actions can be represented as asequence of primitive actions and aspect transitions. For ex-ample, the “flip” macro action is implemented as a sequentialcomposition of the following four primitive actions: 1) reach,2) grasp, 3) lift, and 4) place. These four actions connectfive fine-grained aspect nodes that represent expectations forintermediate observations. Figure 3 shows these intermediatestages. These fine-grained aspect nodes allow us to monitorand detect unplanned transitions at many intermediate stagesof the flip interaction. In Section IV, we describe how failuresare handled in the ATG model.

C. Recursive Bayesian Estimation

In our framework the Bayesian filtering algorithm [24]is used to estimate which aspect node likely generatesthe current observation. The recursive Bayesian estimationdescribed in Figure 4 is used to update the posterior beliefbel over all aspect nodes xt when action at is executed fromstate xt−1 and new observation zt is acquired. The initialbelief is set based on a prior distribution over all aspectnodes.

D. Top Down Inference

The number of different feature types that could exist inan environment could be enormous. A bottom up approachthat runs all possible detectors would be inefficient and ofteninfeasible in robotic applications. Although there is evidencethat human vision is processed in parallel in the primaryvisual cortex [8], there is also evidence that secondary andtertiary cortical areas strongly focus perceptual effort onfeatures that are related to tasks and expectations. We arguethat mid-level perceptual features may be strongly influencedby top down expectations so that only a subset of detectorsmust be active at a time. This is consistent with a hypothesisintroduced by Notredame et al. that explains how visualillusions could be caused by prior expectations overtakingthe actual stimulus [18]. A lot of research in computervision has also been done on building models that performtop down inference [26] [25]. However, when dealing withsingle images independently the ability of top down inferenceis limited. Most work uses information from larger imageregions to infer low level features. In our work, we focus ontop down inference that is based on past observations andhow the actor interacts with objects and the environment.

Fig. 3. Fine-grained flip action. The photos shown from left to right are the five intermediate stages of a flip action. The robot checks whether thesub-action succeeded for each stage.

Given a new set of raw sensor feedback, our frameworkdoes not run all available feature detectors. Instead, onlyfeature detectors corresponding to features in the feature listof an aspect node with significant posterior probability areexecuted. We define a posterior probability to be significantif it is within a fixed percentage difference to the maximumposterior probability among all aspect nodes.

E. Acquiring the Model

In our previous work [14], [15], we demonstrated that ATGmodels can be learned without supervision from a fixed set ofactions and observations. However, with a set of actions thattake continuous parameters, learning ATG models withoutsupervision may require a significant amount of explorationover the parameter space. In [13], we introduced a wayto learn ATG models through imitation learning by tele-operating the robot and memorizing action parameters thattransition between aspect nodes. The work presented in thispaper focuses on how previously acquired ATG models areused to monitor for transition errors during the execution ofa task. Therefore, we employ hand-built ATGs for objects aswell as action for transitioning between aspect nodes. Duringsurprise transitions new models that capture new transitionsare created based on these hand-build models.

IV. HANDLING UNCERTAINTIES

In a non-deterministic environment it is inevitable that arobot will encounter randomness and surprises while per-forming tasks. In this section, we describe approaches tohandling uncertainty in robotics, including a new definitionof surprise.

A. Modeling Randomness

We define a random transition as an outcome of an actionthat is non-deterministic but is within a previously seenset of outcomes that can be captured in the model. Forexample, due to uncertainty, the “flip” macro in Section III-B may place the object in an ungraspable pose and, thus,causing a different observation in the terminating state. Theserandom outcomes are represented in the ATG using transitionprobabilities p(xt|at, xt−1), which is nonzero for all possibleoutcome states. As a consequence, the Bayesian filteringalgorithm will update the belief accordingly. For simplicity,we assign an equal transition probability to all of the outcomestates possible under action at. In general, each outcome ofa randomized transition requires a different action to achievethe goal aspect.

B. Recovery from Surprises

In [3], surprise is described as “the difference between ex-pectation and reality.” Although one might associate surpriseor unexpected events with low probability events, observinga rare event does not always lead to surprise. For example,a person that observes a lottery drawing will likely not besurprised if the outcome is 29 - 41 - 48 - 52 - 54 even thoughthis outcome has a low probability.

Baldi [1] measures the degree of surprise in a Bayesiansetting in which there is a well-defined distribution P (M)over the space of models M. He uses a measure S of howmuch the new information in an observation d changes thedistribution over models. Specifically, he computes the KL-divergence between the current belief state P (M) and thebelief state P (M |d) induced by the new observation:

S(d,M) =

∫MP (M) log(P (M)/P (M |d))dM.

An event is then defined to be a surprise if this divergenceis greater than some threshold θ.

However, this formula considers transitions where aninformative action reduces the entropy over models signifi-cantly as surprising. If there is a uniform distribution overmodels, observing an outcome that concentrates the posteriorprobability on one model should not be surprising. We offer asimpler definition of surprise that agrees better with intuition.Our definition can be applied to a distribution over models,but it also applies when there is just a single model M , andwe start with this simpler case.

The entropy HM (D) = E[− logPM (D)] associated witha model M of observation D defines the expected negativelog probability of an observation D. We define a specific ob-servation d to be surprising if its information, − logPM (d),is much greater than the entropy:

− log(PM (d))� HM (D).

Consider the example of a perfect fair die with six outcomes.No outcome can be surprising since the information of eachoutcome is equal to the entropy of the die. On the other hand,if we acknowledge that there is some minute probability of adie landing on its corner and balancing, then a better estimateof the probability of each face is 1

6−ε, and the probability oflanding on a corner extremely small. If the die does in factland on a corner, it is not the rarity of this event that makesit surprising, but the probability relative to the entropy of the

die. That is, the event is thousands of times less likely thanwhat we expected, which is 1

6 − ε.In the Bayesian setting, where the model M is unknown,

we extend our definition of surprise by simply computingexpectations over the unkown models:

EM[− log(PM (d))]� EM[HM (D)].

Intuitively, a “surprising event” is still an outcome whoseprobability is far lower (and information is much higher)than what was expected. And again, when all events are auniform low probability, there can be no surprise.1

In practice, rather than trying to define a precise probabil-ity for extremely rare events, we simply define events that arenot part of a model as having extremely low probability. Forexample, while the robot is grasping an object if someonecovered the robot’s camera and such a transition is notmodeled in any of the object models, the observation willbe inconsistent with what the robot expects (i.e., has anextremely low probability) and therefore be classified asa surprise transition. During the surprise transition a newATG model that includes this new transition is created byextending the ATG model with the highest belief in memory.If an identical transition reoccurs the robot will be able toexpect the outcome with the new model.

Since surprise transitions are unpredictable and do not liewithin a bounded set of possible outcomes, they are handledby resetting the belief among all aspect nodes to the priordistribution. If a possible plan that will lead to the goal stillexists, the robot will continue to accomplish the task. In ourexperiment, when a surprise transition occurs the robot alsoshows a “What’s up?” gesture that conveys confusion as inFigure 1. This gesture has proven to be useful for indicatingthe robot’s current state in experiments.

V. ALGORITHM

The action selection algorithm that uses the ATG toachieve goals and handle surprise transitions is describedin Figure 5. Here, M represents a set of ATG modelsstored in the robot memory and xtarget is the target aspectgiven to the robot. The function INITIALIZEBELIEF(M)initializes prior probability specified in the set of ATGmodels M. The function INFERASPECTS(M, bel) infersa set of possible aspects Xcandidate based on the beliefbel. The function GETOBSERVATION(Xcandidate) returnsthe current observation zcurrent and the corresponding fea-ture positions in R3 fpose from active feature detectorsderived from the set of possible aspects Xcandidate. Thefunction BAYESFILTEROBS(bel, zcurrent) refers to part ofthe Bayesian filtering algorithm that updates the belief givenan observation. The function SURPRISE(bel) returns trueif the belief in all aspect nodes are zero. If a surprisetransition occurs, the robot shows the “What’s up?” ges-ture, backs up and move arms to the ready pose. The

1This addresses the case of “television snow” that Baldi raises. Whensnow occurs in the context of a low entropy process, it is surprising, sinceit has much lower probability. However, when it occurs in the context of asnow distribution, it has probability equal to the other outcomes.

1: procedure ALGORITHM(xtarget,M)2: bel = INITIALIZEBELIEF(M)3: while true do4: Xcandidate = INFERASPECTS(M, bel)5: zcurrent, fpose = GETOBSERVATION(Xcandidate)6: bel = BAYESFILTEROBS(bel, zcurrent)7: if SURPRISE(bel) then8: WHAT’SUPGESTURE(())9: CREATENEWMODEL(M, bel, a, zcurrent)

10: bel = RESETBELIEF(bel,M)11: BACKUPANDREADYPOSE()12: continue13: end if14: a = INFERACTION(M, xtarget, bel)15: EXECUTEACTION(a, fpose)16: bel = BAYESFILTERACT(bel, a)17: end while18: end procedure

Fig. 5. Algorithm for achieving goal aspect and handling surprisetransitions.

function CREATENEWMODEL(M, bel, a, zcurrent) createsa new ATG model by adding a new aspect node rep-resenting zcurrent and an action edge representing ac-tion a to the ATG model with the highest belief in M.The belief over aspects are reset to the prior distribu-tion through function RESETBELIEF(bel,M). The func-tion INFERACTION(M, xtarget, bel) finds a shortest pathto the target aspect xtarget. The function then returns thefirst action on the shortest path. EXECUTEACTION(a, fpose)executes the action based on information stored in theaction edge a and feature positions fpose. The functionBAYESFILTERACT(bel, a) refers to part of the Bayesianfiltering algorithm that updates the belief given an action.

VI. EXPERIMENTAL RESULTS

In order to evaluate the techniques we introduced to handlerandomness and uncertainty, two different experiments wereperformed. For both cases, the uBot-6 mobile manipulatorhas to perform simple manipulation tasks. The performanceis compared with and without the presented techniques.

A. Experimental Setting

The goal of task is to manipulate an ARcube until therequested features are observed on the corresponding faces.An ARcube is a cube box with a different ARtag on each ofits face as shown in Figure 3. An ARtag is a square fiducialmarker commonly used in robotics due to ease of use androbust detection. In this work an ATG model representing anARcube has 174 nodes and 408 edges.

B. Surprise Recovery

For the first experiment, the model is provided with afairly detailed ATG model of the handled ARcube. Therobot has to reach the desired state with ARtags 1 and 2 onthe top and front faces of the cube respectively. The robothas to first perform this task solely based on the providedmodel and the contained transitions. Despite the modelbeing fairly complete, a small perturbation in the visual and

haptics sensors, as well as slightly different dynamics whenmanipulating the ARcube can result in an observation notcontained in the model or not agreeing with the model (e.g.an action resulting in new outcome). Without the ability torecover, reaching such a state, the robot will not be able tocomplete the task.

The robot was repeatedly provided with random con-figurations of the ARcube and the performance of bothmethods was compared. The robot was able to completethe task reliably 10 out of 10 times when provided withthe ability to recover from unforeseen observations. Withoutthis capability, the robot succeeded 5 out of 10 runs.

C. Error Detection Through Fine-Grained Actions

In the second experiment, the robot was presented with anARcube such that one flip action of the cube will lead to thedesired goal state. The ARcube was then perturbed duringthe grasping action to cause a grasp failure. The performanceof the robot was evaluated based on the number of actionsit needed to complete the task despite this perturbation.

We tested two different setups. In the first setup the robothas access to a “flip” macro which combines four primitiveactions described in Section III-B into one macro action. Inthe second setup, the robot uses the fine-grained actions andaspects directly. The experiment was performed 10 times foreach setup. Using the flip macro, the robot needed 16.1±3.37primitive actions to complete the task, whereas with the fine-grained actions it only needed 10.6± 0.69 primitive actions.The difference in the required number of primitive actions isdue to the capability of detecting the error much earlier andreacting appropriately when using fine-grained actions andaspects.

VII. CONCLUSION

In this work we describe how failures in stochastic roboticactions can be detected and handled properly. We introducea general framework that stores fine-grained event transitionsfor early error detection. We further categorize detectedfailures into random transitions or surprise transitions basedon how it is perceived by the robot. These two types of uncer-tainties are then handled and recovered separately. We haveshown that the overall framework increases the robustnesstowards handling uncertainties and decreases the number ofactions needed on manipulation tasks experimented on theuBot-6 mobile manipulator.

VIII. ACKNOWLEDGMENTThe authors would like to thank Mitchell Hebert, Michael Lanighan,

Tiffany Liu, and Takeshi Takahashi for their contributions. This material isbased upon work supported under Grant NASA-GCT-NNX12AR16A and aNASA Space Technology Research Fellowship. Any opinions, findings, andconclusions or recommendations expressed in this material are those of theauthors and do not necessarily reflect the views of the National Aeronauticsand Space Administration.

REFERENCES

[1] Pierre Baldi. A computational theory of surprise. In Information,Coding and Mathematics, pages 1–25. Springer, 2002.

[2] Daniel E Berlyne. Conflict, arousal, and curiosity. 1960.

[3] John L Casti. Complexification explaining a paradoxical world throughthe science of surprise. 1994.

[4] Bruce R Donald. A geometric approach to error detection and recoveryfor robot motion planning with uncertainty. Artificial Intelligence,37(1):223–271, 1988.

[5] J Gibson. Perceiving, acting, and knowing: Toward an ecologicalpsychology. chap. The Theory of Affordance). Michigan: LawrenceErlbaum Associates, 1977.

[6] Ziv Gigus and Jitendra Malik. Computing the aspect graph forline drawings of polyhedral objects. Pattern Analysis and MachineIntelligence, IEEE Transactions on, 12(2):113–122, 1990.

[7] Harry F Harlow. Learning and satiation of response in intrinsicallymotivated complex puzzle performance by monkeys. Journal ofComparative and Physiological Psychology, 43(4):289, 1950.

[8] DH Hubel, TN Wiesel, and MP Stryker. Orientation columns inmacaque monkey visual cortex demonstrated by the 2-deoxyglucoseautoradiographic technique. Nature, 269(5626):328, 1977.

[9] Clark Leonard Hull. Principles of behavior: an introduction to behaviortheory. 1943.

[10] Laurent Itti and Pierre F Baldi. Bayesian surprise attracts humanattention. In Advances in neural information processing systems, pages547–554, 2005.

[11] Leslie Pack Kaelbling, Michael L Littman, and Anthony R Cassan-dra. Planning and acting in partially observable stochastic domains.Artificial intelligence, 101(1):99–134, 1998.

[12] Jan J Koenderink and Andrea J van Doorn. The internal represen-tation of solid shape with respect to vision. Biological cybernetics,32(4):211–216, 1979.

[13] Li Yang Ku, Erik G Learned-Miller, and Roderic A Grupen. Modelingobjects as aspect transition graphs to support manipulation. Interna-tional Symposium on Robotics Research, 2015.

[14] Li Yang Ku, Shiraj Sen, Erik G Learned-Miller, and Roderic AGrupen. Action-based models for belief-space planning. Workshopon Information-Based Grasp and Manipulation Planning, at Robotics:Science and Systems, 2014.

[15] Li Yang Ku, Shiraj Sen, Erik G Learned-Miller, and Roderic AGrupen. Aspect transition graph: an affordance-based model. Sec-ond Workshop on Affordances: Visual Perception of Affordances andFunctional Visual Primitives for Scene Analysis, at the EuropeanConference on Computer Vision, 2014.

[16] Omid Madani, Steve Hanks, and Anne Condon. On the undecidabilityof probabilistic planning and infinite-horizon partially observablemarkov decision problems. In AAAI/IAAI, pages 541–548, 1999.

[17] Kay C Montgomery. The role of the exploratory drive in learning.Journal of Comparative and Physiological Psychology, 47(1):60,1954.

[18] Charles-Edouard Notredame, Delphine Pins, Sophie Deneve, andRenaud Jardri. What visual illusions teach us about schizophrenia.Frontiers in integrative neuroscience, 8, 2014.

[19] Christos H Papadimitriou and John N Tsitsiklis. The complexityof markov decision processes. Mathematics of operations research,12(3):441–450, 1987.

[20] Pascal Poupart and Craig Boutilier. Bounded finite state controllers. InAdvances in neural information processing systems, page None, 2003.

[21] Alberto Rodriguez, Matthew T Mason, Siddhartha S Srinivasa,Matthew Bernstein, and Alex Zirbel. Abort and retry in grasping. InIntelligent Robots and Systems (IROS), 2011 IEEE/RSJ InternationalConference on, pages 1804–1810. IEEE, 2011.

[22] Dirk Ruiken, Michael W Lanighan, Roderic Grupen, et al. Posturalmodes and control for dexterous mobile manipulation: the umass ubotconcept. In Humanoid Robots (Humanoids), 2013 13th IEEE-RASInternational Conference on, pages 280–285. IEEE, 2013.

[23] Shiraj Sen. Bridging the gap between autonomous skill learningand task-specific planning. PhD thesis, University of MassachusettsAmherst, 2013.

[24] Sebastian Thrun, Wolfram Burgard, and Dieter Fox. Probabilisticrobotics. MIT press, 2005.

[25] Sinisa Todorovic and Narendra Ahuja. Region-based hierarchicalimage matching. International Journal of Computer Vision, 78(1):47–66, 2008.

[26] Tianfu Wu and Song-Chun Zhu. A numerical study of the bottom-up and top-down inference processes in and-or graphs. Internationaljournal of computer vision, 93(2):226–252, 2011.