Adaptive Robot-Assisted Feeding: An Online LearningFramework for Acquiring Previously-Unseen Food Items

Ethan K. Gordon1, Xiang Meng2, Tapomayukh Bhattacharjee1, Matt Barnes1 and Siddhartha S. Srinivasa1

Abstract— A successful robot-assisted feeding system requiresbite acquisition of a wide variety of food items. It needs to adaptto changing user food preferences under uncertain visual andphysical environments. Different food items in different environ-mental conditions may require different manipulation strategiesfor successful bite acquisition. Therefore, a key challenge is tohandle previously-unseen food items with very different successrate distributions over strategy. Combining low-level controllersand planners into discrete action trajectories, we show that theproblem can be represented using a linear contextual banditsetting. We construct a simulated environment using a doubly-robust loss estimate from previously-seen food items, which weuse to tune the parameters of off-the-shelf contextual banditalgorithms. Finally, we demonstrate empirically on a robot-assisted feeding system that, even starting with a model trainedon thousands of skewering attempts on dissimilar previously-seen food items, ε-greedy and LinUCB algorithms can quicklyconverge to the most successful manipulation strategy.

I. INTRODUCTIONEating is an activity of daily living. While many of us

take eating for granted, according to a U.S. study in 2010,approximately 1.0 million people need assistance eating[1]. The ability to self feed would not only save timefor caregivers, but also increase a person’s sense of selfworth [2,3]. While some commercial feeding systems exist[4,5], these systems have minimal autonomy and requirepre-programmed movements, making it difficult for them toadapt to environmental changes. In general, a robust feedingsystem needs to be able to acquire a bite of food in anuncertain environment and transfer it safely to a potentiallyunpredictable user. In this work, we focus on bite acquisition,specifically the acquisition of food items that the robot maynot have seen or manipulated before.

Different food items may require different manipulationstrategies for bite acquisition [6]. While recent work hasachieved some successes in developing strategies that canacquire a variety of food items [7,8], it is unclear whichstrategy works the best on previously-unseen food. Evenfood items that may look similar, such as ripe and un-ripebanana slices, can have very different consistencies leading todifferent bite acquisition strategies. Our key insight is that we

1 Ethan K. Gordon, Tapomayukh Bhattacharjee, Matt Barnes and Sid-dhartha S. Srinivasa are with the Department of Computer Science andEngineering, University of Washington, Seattle, WA 98195 {ekgordon,tapo, mbarnes, siddh}@cs.washington.edu

2Xiang Meng is with the Department of Statistics, University of Wash-ington, Seattle, WA 98195, xmeng051@uw.edu

This work was partially funded by the National Institute of Health R01(#R01EB019335), National Science Foundation CPS (#1544797), NationalScience Foundation NRI (#1637748), the Office of Naval Research, theRCTA, Amazon, and Honda Research Institute USA. We would also liketo thank Joshua Brockschmidt for his help with the robot experiments.

Fig. 1. When faced with new foods, a robot-assisted feeding systemmust decide between a variety of bite acquisition strategies. In our system,each strategy is parameterized by a fork pitch (left: tilted-angle “TA”,vertical “VS”, or tilted-vertical “TV”) and fork roll angle (right: parallel orperpendicular). Our contextual bandit online learning framework learns fromlimited feedback (success or failure) after attempting each bite acquisitionstrategy, which itself depends on perception, planning, and low-level control.

can leverage high-level successful bite acquisition strategiesderived from human user studies [6] and an existing modelbatch-trained on a set of food items to suggest strategysuccess probabilities [7] to perform online learning.

We believe that exploring online learning bite acquisitioncan lead to manipulation strategies that better generalize topreviously-unseen food items. This is due to (a) covariateshift from the training dataset, (b) the diversity of foodcategories and (c) the expensive process of collecting data ona physical robot. Factors which may contribute to covariateshift include changing lighting conditions, backgrounds, andnot knowing the distribution of food items a priori. Anonline learning scheme allows for the system to leverage datacollected in real-world conditions and adapt to each user’sspecific palate.

Importantly, each individual strategy only returns partial(or bandit) feedback. In other words, when our system takesan action to pick up a food item, it can only see whether ithas failed or succeeded with that action. It is not privy tothe counterfactual loss of other actions. Additionally, visualfeatures provide context for each food item. Therefore, theproblem naturally fits into the well-studied contextual banditsetting.

In this work, we propose a contextual bandit frameworkfor this problem setting. We present multiple algorithmsbased on the contextual bandit literature that could providepotential solutions. Our major contributions are (1) thisframework, including a featurizer, simulated hyperparametertuner, integrated off-the-shelf algorithms ε-greedy [9] and

arX

iv:1

908.

0708

8v3

[cs

.RO

] 2

Mar

202

0

Fig. 2. Online learning framework. SPANet is trained on previously-seen food items, and then all but the last layer is frozen as a featurizer. The finallinear layer becomes the “Linear Map” that we update after each subsequent attempt. The result is the estimated success rate of each action on the givenfood item, which we use to select a single action to attempt and update the linear map.

LinUCB [10]; and (2) empirical evidence to its efficacy inreal robot bite acquisition experiments. Our initial actionspace of 3 fork roll angles (tilted-angled (TA), vertical (VS),and tilted-vertical (TV), as described in Figure 1) × 2 forkpitch angles currently limit us to discrete, solid food items,but future work can examine a richer action space to tacklebite acquisition on even more varied food items and realisticplates.

II. RELATED WORK

A. Robot-Assisted Feeding: Food Manipulation

Food manipulation has been studied for various applica-tions such as the packaging industry [11–16] with focuson the design of application-specific grippers for robustsorting and pick-and-place, as well as showing the needfor visual sensing for quality control [17–19] and hapticsensing for grasping deformable food items without dam-aging them [11–16]. Research labs have also explored mealpreparation [20,21] as an exemplar multi-step manipulationproblem, baking cookies [22], making pancakes [23], sep-arating Oreos [24], and preparing meals [25] with robots.Most of these studies either interacted with a specific fooditem with a fixed manipulation strategy [22,23] or used a setof food items for meal preparation which required a differentset of manipulation strategies [25].

Existing autonomous robot-assisted feedingsystems [7,8,26,27] can acquire a fixed set of fooditems and feed people, but it is not clear whether thesesystems can adapt to very different food items that requirecompletely different strategies. Feng et al. [7] developedthe Skewering Position Action Network (SPANet) and showgeneralization to previously-unseen food items, but only forthose with similar bite acquisition strategies. The universeof food items is massive and thus, it is almost impossible totrain these systems on every kind of food items that exist.And even if we could, as a static model, it is still vulnerableto the covariate shift discussed in Section I. Our paperaims to address this gap in the food manipulation literatureby developing methods that can generalize to previously-unseen food items with very different action distributions.

We propose to use an online learning framework in acontextual bandit setting for food manipulation.

B. Online Learning

Bandit algorithms have seen widespread success in onlineadvertising [28,29], health interventions [30,31], clinicaltrials [32], adaptive routing [33], education [34], musicrecommendations [35], financial portfolio design [36], orany application requiring a more optimized version of A/Btesting. Adoption in robotics has been more limited, such astrajectory selection for object rearrangement planning [37],kicking strategies in robotic soccer [38], and perhaps mostclosely related, selecting among deformable object modelsfor acquisition tasks [39]. Unlike previous work, we argueit is untennable to construct deformable object models forevery food item, as conventional grocery stores typicallystock in excess of 40,000 products [40]. Instead, we takea model-free approach which operates directly on the imagecontext space.

No-regret algorithms for solving bandit problems includeUCB [41] and EXP3 [42] for stochastic and adversarialreward distributions, respectively. This was also extended tothe bandits-with-expert-advice setting (a generalization of thecontextual bandit problem for small policy classes) in EXP4[42]. Baseline methods for the contextual bandit probleminclude epoch-greedy [43] and greedy [44], both of which aresimple to implement, perform well in practice, although donot achieve optimal regret guarantees. More recent advancesinclude LinUCB [45], RegCB [46] and Online Cover [47],a computationally efficient approximation to an algorithmwhich achieves optimal regret. For a recent and thoroughoverview, we refer the interested reader to [9,48].

III. ONLINE LEARNING WITH CONTEXTUALBANDITS

A. Motivation

As discussed in Section II-A, even when we controlfor covariate shift in a laboratory setting and switch toan unbiased success rate estimate (see Figure 3), SPANetis unable to generalize to some previously-unseen foodcategories (specifically kiwi and banana). We hypothesizethat this lack of generalization is due partly to the high

kiw

i

broc

coli

stra

wbe

rry

caul

iflow

er

spin

ach

grap

e

hone

ydew

bana

na

0

0.25

0.5

0.75

1

Food Item Excluded from Training Set

Succ

ess

Rat

eEs

timat

e

In-ClassRandom

Fig. 3. Generalization results for SPANet on select food items usingdata from [7] and the unbiased estimator described in Section IV-A. Whenexcluded from the training set, each item performs worse, with banana inparticular performing significantly worse than even random (p < 0.05).

diversity of actions for these food categories. For example,the most successful fork pitch for kiwi and banana was TA,which is very different from the successful actions for therest of the food item dataset. To determine whether collectingadditional training data would solve this problem, we con-trolled for both the number of previously-seen food classesand the total number of previously-seen training examples.The results, shown in Figure 4, do not show a noticeableimprovement in out-of-class performance. An online learnapproach allows for training to continue indefinitely, bringout-of-class food items into the effective training set. It alsoeffectively amortizes the potentially time-consuming processof data collection (SPANet’s dataset, for example, requiredapproximately 81 hours of supervision) over the useful lifeof the system.

B. Formulation

A general contextual bandit algorithm consists of twoparts. (1) An exploration strategy determines which actionto take at each time step given the context and some policy.(2) A learner incorporates the bandit feedback received eachtime step into the policy. This structure as it applies in thisenvironment with SPANet features is presented in Algorithm1.

At each round t = 1, . . . , T , the interaction protocolconsists of

1) Context observation The user selects a food item toacquire (in this work, we use RetinaNet [49] to detectobjects). We observe an RGBD image containing thesingle food item. We pass this through SPANet (Sec-tion II-A) and use the penultimate layer as the contextfeatures xt ∈ Rd.

2) Action selection The algorithm selects one manipula-tion strategy at ∈ A = {1, 2, . . . ,K}. In our initialimplementation, K = 6, with 3 pitch angles (VS, TV,TA) and 2 roll angles (parallel and perpendicular tothe food), as described in Figure 1.

3) Partial loss observation The environment provides abinary loss ct(at, xt) ∈ {0, 1}, where ct = 0 corre-sponds to the robot successfully acquiring the singledesired food item.

25% 50% 75% 100%0

0.25

0.5

0.75

1

Avg.

Str.Bro.

Car.

Cel.

Hon.

Ban.

Amount of Training Data Used(0% = Random Action)

SPA

Net

Succ

ess

Rat

e

8 12 150

0.25

0.5

0.75

1

Avg.

Str.Bro.

Hon.

Ban.

Kiw.

Spi.

Gra.Cau.

Number of Classes in Training Set(0 = Random Action)

SPA

Net

Succ

ess

Rat

eFig. 4. SPANet out-of-class success rate using data from [7], given differentamounts of data and food classes included in the training dataset. Each blackline represents a single food item excluded from the training set. The redline represents the performance averaged across all food items. The amountof out-of-class training data has already reached the point of diminishingreturns at best. For very different food items (like banana), the extra dataactually reduces performance, likely due to overfitting to the fixed set offood classes.

A flow diagram of this protocol and its components ispresented in Figure 2.

The algorithm itself consists of a stochastic policy π(xt) =P(at = a|xt), and the goal is to minimize the cumulativeregret of this policy. In other words, we wish to minimizeRT , which is the difference in performance between ourpolicy π and the best possible policy π∗ ∈ Π for the lifetimeof our program T . With ct ∈ C, xt ∈ X , at ∈ A at time t:

RT :=

T∑t=1

ct(π(φ(xt)))−T∑t=1

ct(π∗(φ(xt))) (1)

In cases where we compare algorithms with different setsΠ, such as when tuning on the dimensionality d as a hy-perparameter, we shall instead look to minimize cumulativeloss, the first term of RT .

C. Learning: Importance-Weighted Linear Regression

The learning portion of a contextual bandit algorithmoperates by first using past observations to estimate the costof all actions for a given context. This reduces the prob-lem to off-policy supervised learning. Since the contextualbandit literature tends to focus on exploration strategy, thesub-algorithm which performs the underlying full-feedbackclassification or regression is referred to as an oracle. Allalgorithms we use here will use an importance-weightedlinear regression oracle.

For our feature extractor, we are using the activations ofthe penultimate layer in SPANet, and fine tune the final

Algorithm 1: General Contextual Bandit with SPANetFeatures

Input: Trained SPANet φ, Environment EInitialize Context x ∈ X ∼ Efor t = 1, . . . , T do

Find features φ(x)pt ← explore(φ(x))Select action at ∼ ptReceive ct ∼ E|atlearn(φ(x), at, ct, pt)if ct = 0 then

Re-sample context x ∼ Eend

end

Algorithm 2: Importance-Weighted Regression OracleInput: Regularization parameter λ, d (features)Initialize π0: ∀a ∈ A: Aa ← λId×d; ba ← 0Function oracle(π, φ(x), at, ct, pt(at)):

(A, b)← πAat ← Aat + 1

ptφφT

bat ← bat + ctptφ

θat ← A−1at batπ′ ← (θ, A, b)

return

layer in an online fashion. Thus, justified by the success ofSPANet, we assume a linear map from the Rd features to theexpected cost of each action: E[ct|at, xt] = θTatφ(xt). In thiscase, the regression oracle computes a weighted least-squaresestimate θ:

θ :=

T∑t=0

1

pt(at)

(θTatφ(xt)− ct(at)

)2(2)

Similarly to inverse propensity-scoring [47], the weightpt(at)

−1 ensures that this returns an unbiased estimate ofthe underlying true weights θ∗. An implementation of thisoracle is shown in Algorithm 2. The policy associated witha given weight estimate θ will just be the greedy policy:πθ(x) = arg mina θ

Ta φ(x)

D. Exploration Strategy: ε-greedy

One of the simplest approaches to exploration is the ε-greedy algorithm, as shown in Algorithm 3. This algorithmopts for the optimal action based on previous observationswith probability (1 − ε) and explores all actions uniformlywith probability ε. We consider both purely greedy (ε = 0)and exploratory (ε > 0) variants.

With arbitrary contexts, the ε-greedy algorithm (with op-timized ε) has a cumulative regret bound RT ∼ O(T 2/3),though it can perform well empirically [9]. The fact thatcontexts can be repeated on failure may also allow for abetter regret bound, as taking multiple actions can provideeffectively better-than-bandit feedback for a given context.

Algorithm 3: ε-greedyInput: Exploration parameter ε ∈ [0, 1)Function explore(φ(x)):

pt(a)← εK + (1− ε)1{πt(φ(x))}

returnFunction learn(φ(x), at, ct, pt):

πt+1 = oracle(πt, φ(x), at, ct, pt(at))return

Algorithm 4: LinUCBInput: Width parameter αFunction explore(φ(x)):

for a ∈ A do

ucba ← θTa φ(x) + α√φ(x)TA−1a φ(x) [10]

endreturnFunction learn(φ(x), at, ct, pt):

πt+1 = oracle(πt, φ(x), at, ct, pt(at))return

E. Exploration Strategy: LinUCB

The other algorithm we use is Linear Upper Confidencebound (LinUCB), as presented in Algorithm 4. We justify theuse of LinUCB [50] due to the linear form of the ultimateSPANet layer (as justified in Section III-C). Unlike ε-greedy,the regret bound for LinUCB holds even if an adversarywas choosing the worst-case contexts to show. Therefore,LinUCB can in theory be robust against covariate shift,allowing it to potentially be very competitive in this setting.

At each time step, we choose the action that maximizesthe reward UCB (or, equivalently, loss LCB). This implicitlyencourages exploration. In a choice between two actionswith similar expected cost, the algorithm will opt for theone with higher variance. With arbitrary contexts, LinUCBhas a cumulative regret bound RT ∼ O(T 1/2), which is animprovement over ε-greedy in the worst case. Like ε-greedy,seeing repeated contexts on failure may improve this bound.

IV. EXPERIMENTS

A. Tuning in Simulation

In addition to the normal hyperparameters associated withlinear regression (dimensionality d and L2 regularizer λ),each algorithm has its own exploration hyperparameter. Wetune these by constructing a simulated training environmentusing the data from [7]. Specifically, we exclude fromSPANet three food items with very different success ratedistributions over strategies. Banana is very sensitive to forkpitch, with TA performing the best by a wide margin, asit prevents the slice from slipping off the fork. Grape is,in general, very difficult to pick up, with the best strategydependent on the biases in perception and planning. Appleis, in general, very easy to pick up, with some sensitivity to

10−3 10−2 10−1 100 101 102 103 104

−0.2−0.1

00.10.2

λ

E[π

∗Lo

ss]

λ Tuning, d = 2048

0 500 1,000 1,5000

200

400

600

Number of Simulated Attempts

Cum

ulat

ive

Loss

ε-greedy Tuning, d = 2048, λ = 100Greedy ε = 0.1 ε = 0.2ε = 0.3 ε = 0.4 ε = 0.5

0 500 1,000 1,5000

200

400

600

Number of Simulated Attempts

Cum

ulat

ive

Loss

LinUCB Tuning, d = 2048, λ = 100α = 0.001 α = 0.005 α = 0.01α = 0.05 α = 0.1 α = 0.5

Fig. 5. Hyperparameter tuning in simulation, with banana, apple, and grape excluded from SPANet. Left shows π∗ performance as a function of λ with95% confidence intervals. Center/Right show the cumulative loss of the contextual bandit algorithms on the excluded food items as a function of theirexploration hyperparameters. On this dataset, LinUCB exhibits more stable competitive performance than ε-greedy.

15 30 45 600

5

10

15

Acquisition Attempt

Cum

ulat

ive

Loss

Greedyε = 0.1

LinUCB (α = 0.01)

Fig. 6. Results of Experiment 1 (Right) using the Autonomous Dexterous Arm (ADA) (Left). SPANet was trained on 12 food types, excluding apple,banana, and grape, 3 types of food with significantly different success rate distributions over strategy. Initialized to θ0 = ~0, the robot cycled 20 timesthrough all 3 food types. For each algorithm, the vertical line represents the point after which 100% of the strategies selected were among the best strategiesfor the food item observed. Greedy is competitive since there is no pre-training to weigh it down (as in Experiment 2), and ε-greedy can still choose abad strategy after convergence. Regardless, all algorithms converge within ∼ 10 failures per food item.

fork roll angle, as it is a long food item. Based on [7], VS,perpendicular roll angle, is likely the best by a slight margin.

Since this data, by necessity, was collected with banditfeedback, the original work imputed the full loss vector ofeach context by averaging the success rate of a given actionacross all food items of the same type. While simple, this canintroduce a herding bias into the simulation relative to thereal world. We eliminate bias in our dataset using a doubly-robust [51] estimator:

lDR(xi, a) = la + (li − la)1(ai = a)

p(ai|xi)(3)

where la is the imputed value from herding, p(ai|xi) is theprobability that we took action ai during data collection( 16 in our case, since data was collected uniformly acrossall actions), and li the actual binary loss associated withthat sample (only available for ai). This estimator eliminatesbias (i.e., E[lDR] = l) from our imputed values at the costof added variance. For each set of hyperparameters, π∗ isdetermined by performing full-feedback least-squares linearregression on all previously-unseen food items to estimateθ∗.

First, we tuned the linear regression parameters d andλ. Using the original SPANet feature space of R2048, wefound that we needed significant regularization (large λ)to see any results on our limited dataset. However, whilereducing our feature-space dimensionality d could in theoryimprove our regret bounds (e.g., LinUCB’s RT ∼ O(d)),it empirically reduced our best possible (π∗) performance.This exposed us to a two-dimensional trade-off of bias vs.variance and performance vs. data-efficiency. For d, the hit

to π∗ outweighs any improvements in regret. For λ, while100 and 1000 produced similar π∗ performance (as seenin Figure 5a), λ = 100 had better performance on greedycumulative loss.

Figure 5b,c shows the results of tuning the explorationparameters ε and α. Note that a greater loss is expected,as the doubly-robust losses have a higher variance thanreality. Stochastic ε-greedy showed a clear local minimum atε = 0.1. Meanwhile, LinUCB demonstrated more consistentcompetitive performance across multiple orders of magnitudefor α. We selected α = 0.01, which reached a slightminimum loss, for the real robot experiments.

B. Real Robot Experiments

a) System Description: Our setup, the AutonomousDexterous Arm (ADA) (Figure 6, left), consists of a 6 DoFJACO robotic arm [52]. The arm has 2 fingers that grab aninstrumented fork (forque) using a custom-built, 3D-printedfork holder. The system uses visual and haptic modalities toperform the feeding task. For haptic input, we instrumentedthe forque with a 6-axis ATI Nano25 Force-Torque sensor[53]. We use haptic sensing to control the end effector forcesduring skewering. For visual input, we mounted a custombuilt wireless perception unit on the robot’s wrist; the unitincludes the Intel RealSense D415 RGBD camera and theNVidia Jetson Nano for wireless transmission. Food is placedon a plate mounted on an anti-slip mat commonly found inassisted living facilities.

b) General Procedure: For each trial, we place a singlefood item in the center of the plate. ADA positions itselfvertically above the plate and performs object detection and

20 25 300

5

10

15

20

Banana Acquisition Attempt

Cum

ulat

ive

Loss Greedy

ε = 0.1LinUCB (α = 0.01)

20 25 300

5

10

15

20

Carrot Acquisition Attempt

Cum

ulat

ive

Loss Greedy

ε = 0.1LinUCB (α = 0.01)

Fig. 7. Empirical cumulative loss for each contextual bandit algorithm on an unseen food item when θ0 was trained on all previously-seen food items.For each experiment, trials 20-25 were performed on a different, previously-seen food item to demonstrate that the policy did not forget its best strategyduring the unseen food trials. LinUCB demonstrates the most stable performance, especially when there are multiple good strategies (as in carrot).

VS TV TA0

0.25

0.5

0.75

1

Action

Loss

Est

imat

e(U

CB

)

Banana Trial 1

VS TV TA0

0.25

0.5

0.75

1

Action

Banana Trial 10

VS TV TA0

0.25

0.5

0.75

1

Action

Banana Trial 20

VS TV TA0

0.25

0.5

0.75

1

Action

Banana Trial 30

Fig. 8. Evolution of the internal upper confidence bound (UCB) estimate of the success rate of each action over time for the banana trials. Since bananais rotationally symmetric, for each fork pitch, only the maximum UCB of the two roll angles is presented.

featurization using a checkpoint of SPANet that has beentrained with some food items excluded. Importantly, theidentity of the food items, while used for object detection,was never made available to the contextual bandit algorithm.After performing the requested action, the binary loss isrecorded manually and the learning algorithm is updated.To mimic a realistic feeding setting, we only removed andreplaced the food item after a successful acquisition.

We define a bite acquisition attempt as a success (ct = 0)if the target food item, either the whole piece or a cut portionremains on the fork for 5 seconds after removal from theplate. If the target food item is skewered with at least 2 outof 4 tines but the fork fails to pick it up or the food falls offsoon after lift-off, the attempt is deemed a failure (ct = 1).If less than 2 out of 4 tines touch a food item due to system-level errors (e.g., perception or planning), we discard the trialcompletely.

c) Experiment 1: The purpose of this first experimentis to determine whether the features generated by SPANettrained without previously-unseen food items are rich enoughfor the contextual bandit algorithm to find the best strategyfor multiple food items. We cycle through 3 food items,apple, banana, then grape, 20 times, leading to 60 total trials.We choose these items for the same reason as the simulation:they are representative of the majority of our food dataset.

d) Experiment 2: This experiment tests whether thecontextual bandit algorithms can adapt to new food itemswhen given a θ that has already been trained on manypreviously-seen dissimilar food items from the doubly-robustsimulated environment. Unlike Experiment 1, we only teston one food item at a time, so that the set of dissimilar fooditems is of a non-negligible size. For banana, θ was trained

on all ∼ 8000 attempts on all 15 non-banana food items,at is the only food item sensitive to fork pitch where TA isthe best strategy. For carrot, which is very sensitive to forkroll (i.e., VS and TV, perpendicular roll angle, are the likelybest strategies by a wide margin), θ was trained on ∼ 3000attempts that excluded other food items sensitive to fork roll,such as apple, bell pepper, and celery. For each food item, weconducted 20 trials, followed by 5 trials with a previously-seen food item (grape and banana, respectively), followed byanother 5 trials of the test food item, to ensure that π didnot forget previously-seen food items after adapting to a newfood item.

V. RESULTS

The results of Experiment 1 are summarized in Figure 6(right). All algorithms suffered a cumulative loss between 10and 15. The key takeaway is that all algorithms convergedto the best strategy set within ∼ 10 failures per new fooditem, after which, the best strategy (or a strategy withinthe best set of strategies) was chosen 100% of the timefor each food item. The only errors after this point weredue to uncertainties in perception and planning. Interestingly,greedy had the highest performance by this metric, thoughunlike Experiment 2, it was not weighed down by pre-training in θ, and greedy is often empirically competitive incontextual bandit settings [9]. These results suggest that theSPANet features are indeed rich enough for the contextualbandit algorithms to learn the best strategy for multiplerepresentative food items simultanesously.

The results of Experiment 2 are summarized in Figure 7.LinUCB exhibited superior cumulative loss performance forboth food items, and greedy exhibited particularly poor per-

formance. This inverse to Experiment 1, this is probably dueto the weight of the pre-trained θ forcing greedy to try pre-viously good strategies before exploring new ones. LinUCBcould capitalize on the uncertainty introduced by seeing asignificantly different context. Figure 8 shows how LinUCB’supper confidence bound estimates changed over time as itadapted to banana. Regardless, the consistent performanceon the previously-seen food item did demonstrate that thecontextual bandit algorithm could adapt to new informationwithout forgetting the best strategis of previously-seen fooditems.

VI. DISCUSSION

One key takeaway from these results is that LinUCB isempirically robust across a range of hyper-parameters andinitializations. A fluke early failure will not sink a high-expectation action, as increasing variance dampens the de-creasing expectation. Robustness is vital for a robotic feedingsystem, as users, especially those with some mobility, maynot tolerate many errors in an autonomous system they maywant to use daily [54]. While the number of failures seen heremay be a lot for a single meal, both experiments suggest thatthis is a 1-time cost that can be amortized over the life ofthe feeding system.

In future work, we intend to broaden our scope to multiplefood items by considering the entire plate of food items asa single compound state, or just switching food items if theexpected success rate of all actions is below some threshold.

Beyond using RGBD context features, our robot hasaccess to other modalities, including haptic feedback. Non-destructive probing can provide us a richer context, especiallyif we need to differentiate between similar-looking fooditems with different material properties (say, because oneis cooked or ripe). Other groups have found success using avibration-detecting audio modality [55] as well.

Finally, we only investigated discrete, solid food items.In order to generalize to a realistic average plate withcontinuous and mixed foods, we will need to expand to aricher action space. Since adding more action parameters(e.g. roll, pitch, yaw) will increase the size of the actionspace at a combinatorial rate, we could leverage similaritiesbetween actions by modeling each one as a coupled slate ofactions [56].

Overall, these results suggest that a contextual banditapproach with discrete, dissimilar actions is a promisingroute to data-efficient adaptive bite acquisition.

REFERENCES

[1] M. W. Brault, “Americans with disabilities: 2010,” Current populationreports, vol. 7, pp. 70–131, 2012.

[2] S. D. Prior, “An electric wheelchair mounted robotic arm-a survey ofpotential users,” Journal of medical engineering & technology, vol. 14,no. 4, pp. 143–154, 1990.

[3] C. A. Stanger, C. Anglin, W. S. Harwin, and D. P. Romilly, “Devicesfor assisting manipulation: a summary of user task priorities,” IEEETransactions on rehabilitation Engineering, vol. 2, no. 4, pp. 256–265,1994.

[4] MySpoon, 2018, https://www.secom.co.jp/english/myspoon/food.html.[5] Obi, 2018, https://meetobi.com/.[6] T. Bhattacharjee, G. Lee, H. Song, and S. S. Srinivasa, “Towards

robotic feeding: Role of haptics in fork-based food manipulation,”IEEE Robotics and Automation Letters, 2019.

[7] R. Feng, Y. Kim, G. Lee, E. K. Gordon, M. Schmittle, S. Kumar,T. Bhattacharjee, and S. S. Srinivasa, “Robot-assisted feeding: Gener-alizing skewering strategies across food items on a realistic plate,” inInternational Symposium on Robotics Research, 2019.

[8] D. Gallenberger, T. Bhattacharjee, Y. Kim, and S. Srinivasa, “Transferdepends on acquisition: Analyzing manipulation strategies for roboticfeeding,” ACM/IEEE International Conference on Human-Robot In-teraction, 2019.

[9] A. Bietti, A. Agarwal, and J. Langford, “A contextual bandit bake-off,”arXiv preprint 1802.04064, Feb. 2018.

[10] C. Wei, L. Li, L. Reyzin, and R. E. Schapire, “Contextual bandits withlinear payoff functions,” in Proceedings of the Fourteenth InternationalConference on Artificial Intelligence and Statistics, 2011, pp. 208–214.

[11] P. Chua, T. Ilschner, and D. Caldwell, “Robotic manipulation offood products–a review,” Industrial Robot: An International Journal,vol. 30, no. 4, pp. 345–354, 2003.

[12] F. Erzincanli and J. Sharp, “Meeting the need for robotic handling offood products,” Food Control, vol. 8, no. 4, pp. 185–190, 1997.

[13] R. Morales, F. Badesa, N. Garcia-Aracil, J. Sabater, and L. Zollo, “Softrobotic manipulation of onions and artichokes in the food industry,”Advances in Mechanical Engineering, vol. 6, p. 345291, 2014.

[14] P. Brett, A. Shacklock, and K. Khodabendehloo, “Research towardsgeneralised robotic systems for handling non-rigid products,” in Inter-national Conference on Advanced Robotics. IEEE, 1991, pp. 1530–1533.

[15] T. G. Williams, J. J. Rowland, and M. H. Lee, “Teaching fromexamples in assembly and manipulation of snack food ingredients byrobot,” in IEEE/RSJ International Conference on Intelligent Robotsand Systems, vol. 4. IEEE, 2001, pp. 2300–2305.

[16] C. Blanes, M. Mellado, C. Ortiz, and A. Valera, “Technologies forrobot grippers in pick and place operations for fresh fruits andvegetables,” Spanish Journal of Agricultural Research, vol. 9, no. 4,pp. 1130–1141, 2011.

[17] T. Brosnan and D.-W. Sun, “Inspection and grading of agriculturaland food products by computer vision systems–a review,” Computersand Electronics in Agriculture, vol. 36, no. 2, pp. 193–213, 2002.

[18] C.-J. Du and D.-W. Sun, “Learning techniques used in computer visionfor food quality evaluation: a review,” Journal of food engineering,vol. 72, no. 1, pp. 39–55, 2006.

[19] K. Ding and S. Gunasekaran, “Shape feature extraction and classifi-cation of food material using computer vision,” Transactions of theASAE, vol. 37, no. 5, pp. 1537–1545, 1994.

[20] W.-T. Ma, W.-X. Yan, Z. Fu, and Y.-Z. Zhao, “A chinese cookingrobot for elderly and disabled people,” Robotica, vol. 29, no. 6, pp.843–852, 2011.

[21] Y. Sugiura, D. Sakamoto, A. Withana, M. Inami, and T. Igarashi,“Cooking with robots: designing a household system working in openenvironments,” in Proceedings of the SIGCHI Conference on HumanFactors in Computing Systems. ACM, 2010, pp. 2427–2430.

[22] M. Bollini, J. Barry, and D. Rus, “Bakebot: Baking cookies with thepr2,” in The PR2 workshop: results, challenges and lessons learnedin advancing robots with a common platform, IROS, 2011.

[23] M. Beetz, U. Klank, I. Kresse, A. Maldonado, L. Mosenlechner,D. Pangercic, T. Ruhr, and M. Tenorth, “Robotic roommates mak-ing pancakes,” in IEEE-RAS International Conference on HumanoidRobots. IEEE, 2011, pp. 529–536.

[24] “Oreo separator machines,” https://vimeo.com/63347829,[Online; Re-trieved on 1st February, 2018].

[25] M. C. Gemici and A. Saxena, “Learning haptic representation formanipulating deformable food objects,” in IEEE/RSJ InternationalConference on Intelligent Robots and Systems. IEEE, 2014, pp. 638–645.

[26] D. Park, Y. K. Kim, Z. M. Erickson, and C. C. Kemp, “Towardsassistive feeding with a General-Purpose mobile manipulator,” arXivpreprint arXiv:1605.07996, May 2016.

[27] L. V. Herlant, “Algorithms, Implementation, and Studies on Eatingwith a Shared Control Robot Arm,” Ph.D. dissertation, The RoboticsInstitute Carnegie Mellon University, 2016.

[28] L. Tang, R. Rosales, A. Singh, and D. Agarwal, “Automatic ad formatselection via contextual bandits,” in Proceedings of the 22nd ACMinternational conference on Information & Knowledge Management.ACM, 2013, pp. 1587–1594.

[29] L. Bottou, J. Peters, J. Quinonero-Candela, D. X. Charles, D. M.Chickering, E. Portugaly, D. Ray, P. Simard, and E. Snelson, “Counter-factual reasoning and learning systems: The example of computationaladvertising,” The Journal of Machine Learning Research, vol. 14,no. 1, pp. 3207–3260, 2013.

[30] P. Klasnja, E. B. Hekler, S. Shiffman, A. Boruvka, D. Almirall,A. Tewari, and S. A. Murphy, “Microrandomized trials: An experimen-tal design for developing just-in-time adaptive interventions.” HealthPsychology, vol. 34, no. S, p. 1220, 2015.

[31] I. Hochberg, G. Feraru, M. Kozdoba, S. Mannor, M. Tennenholtz, andE. Yom-Tov, “Encouraging physical activity in patients with diabetesthrough automatic personalized feedback via reinforcement learning

improves glycemic control,” Diabetes care, vol. 39, no. 4, pp. e59–e60, 2016.

[32] S. M. Shortreed, E. Laber, D. J. Lizotte, T. S. Stroup, J. Pineau, andS. A. Murphy, “Informing sequential clinical decision-making throughreinforcement learning: an empirical study,” Machine learning, vol. 84,no. 1-2, pp. 109–136, 2011.

[33] B. Awerbuch and R. D. Kleinberg, “Adaptive routing with end-to-end feedback: Distributed learning and geometric approaches,” inProceedings of the thirty-sixth annual ACM symposium on Theoryof computing. ACM, 2004, pp. 45–53.

[34] T. Mandel, Y.-E. Liu, S. Levine, E. Brunskill, and Z. Popovic, “Offlinepolicy evaluation across representations with applications to educa-tional games,” in Proceedings of the 2014 international conference onAutonomous agents and multi-agent systems. International Foundationfor Autonomous Agents and Multiagent Systems, 2014, pp. 1077–1084.

[35] X. Wang, Y. Wang, D. Hsu, and Y. Wang, “Exploration in inter-active personalized music recommendation: a reinforcement learningapproach,” ACM Transactions on Multimedia Computing, Communi-cations, and Applications (TOMM), vol. 11, no. 1, p. 7, 2014.

[36] W. Shen, J. Wang, Y.-G. Jiang, and H. Zha, “Portfolio choiceswith orthogonal bandit learning,” in Twenty-Fourth International JointConference on Artificial Intelligence, 2015.

[37] M. C. Koval, J. E. King, N. S. Pollard, and S. S. Srinivasa, “Robusttrajectory selection for rearrangement planning as a multi-armed banditproblem,” in 2015 IEEE/RSJ International Conference on IntelligentRobots and Systems (IROS). IEEE, 2015, pp. 2678–2685.

[38] J. P. Mendoza, R. Simmons, and M. Veloso, “Online learning ofrobot soccer free kick plans using a bandit approach,” in Twenty-SixthInternational Conference on Automated Planning and Scheduling,2016.

[39] D. McConachie and D. Berenson, “Bandit-based model selection fordeformable object manipulation,” arXiv preprint arXiv:1703.10254,2017.

[40] A. Malito, “Grocery stores carry 40,000 more items than they did inthe 1990s,” 2017.

[41] P. Auer, N. Cesa-Bianchi, and P. Fischer, “Finite-time analysis of themultiarmed bandit problem,” Machine learning, vol. 47, no. 2-3, pp.235–256, 2002.

[42] P. Auer, N. Cesa-Bianchi, Y. Freund, and R. E. Schapire, “The non-stochastic multiarmed bandit problem,” SIAM journal on computing,vol. 32, no. 1, pp. 48–77, 2002.

[43] J. Langford and T. Zhang, “The epoch-greedy algorithm for multi-

armed bandits with side information,” in Advances in neural informa-tion processing systems, 2008, pp. 817–824.

[44] H. Bastani, M. Bayati, and K. Khosravi, “Mostly exploration-freealgorithms for contextual bandits,” arXiv preprint arXiv:1704.09011,2017.

[45] L. Li, W. Chu, J. Langford, and R. E. Schapire, “A Contextual-Bandit approach to personalized news article recommendation,” arXivpreprint arXiv:1003.0146, Feb. 2010.

[46] D. J. Foster, A. Agarwal, M. Dudık, H. Luo, and R. E. Schapire,“Practical contextual bandits with regression oracles,” arXiv preprintarXiv:1803.01088, Mar. 2018.

[47] A. Agarwal, D. Hsu, S. Kale, J. Langford, L. Li, and R. E. Schapire,“Taming the monster: A fast and simple algorithm for contextualbandits,” arXiv preprint arXiv:1402.0555, Feb. 2014.

[48] T. Lattimore and C. Szepesvari, Bandit Algorithms. CambridgeUniversity Press, 2019.

[49] T. Lin, P. Goyal, R. Girshick, K. He, and P. Dollar, “Focal loss fordense object detection,” in 2017 IEEE International Conference onComputer Vision (ICCV), Oct. 2017, pp. 2999–3007.

[50] Y. Abbasi-yadkori, D. Pal, and C. Szepesvari, “Improved algorithmsfor linear stochastic bandits,” in Advances in Neural InformationProcessing Systems 24, J. Shawe-Taylor, R. S. Zemel, P. L. Bartlett,F. Pereira, and K. Q. Weinberger, Eds. Curran Associates, Inc., 2011,pp. 2312–2320.

[51] M. Dudik, J. Langford, and L. Li, “Doubly robust policy evaluationand learning,” arXiv preprint arXiv:1103.4601, Mar. 2011.

[52] K. JACO, 2018, https://www.kinovarobotics.com/en/products/robotic-armseries.

[53] A.-I. F.-T. Sensor, 2018, https://www.ati-ia.com/products/ft/ft models.aspx?id=Nano25.

[54] T. Bhattacharjee, E. Gordon, R. Scalise, M. Cabrera, A. Caspi,M. Cakmak, and S. Srinivasa, “Is more autonomy always better?exploring preferences of users with mobility impairments in robot-assisted feeding,” in ACM/IEEE International Conference on Human-Robot Interaction, 2020.

[55] S. Clarke, T. Rhodes, C. G. Atkeson, and O. Kroemer, “Learningaudio feedback for estimating amount and flow of granular material,”in Proceedings of The 2nd Conference on Robot Learning, ser.Proceedings of Machine Learning Research, A. Billard, A. Dragan,J. Peters, and J. Morimoto, Eds., vol. 87. PMLR, 2018, pp. 529–550.

[56] M. Dimakopoulou, N. Vlassis, and T. Jebara, “Marginal posteriorsampling for slate bandits,” in Proceedings of the Twenty-EighthInternational Joint Conference on Artificial Intelligence, 2019, pp.2223–2229.