Natural grasp intention recognition based on gaze fixation inhuman-robot interaction

Bo Yang, Student Member, IEEE, Jian Huang∗, Senior Member, IEEE, Xiaolong Li,Xinxing Chen, Student Member, IEEE, Caihua Xiong, Member, IEEE and Yasuhisa Hasegawa Member, IEEE

Abstract— Eye movement is closely related to limb actions, soit can be used to infer movement intentions. More importantly,in some cases, eye movement is the only way for paralyzedand impaired patients with severe movement disorders tocommunicate and interact with the environment. Despite this,eye-tracking technology still has very limited application sce-narios as an intention recognition method. The goal of thispaper is to achieve a natural fixation-based grasping intentionrecognition method, with which a user with hand movementdisorders can intuitively express what tasks he/she wants to doby directly looking at the object of interest. Toward this goal,we design experiments to study the relationships of fixations indifferent tasks. We propose some quantitative features fromthese relationships and analyze them statistically. Then wedesign a natural method for grasping intention recognition. Theexperimental results prove that the accuracy of the proposedmethod for the grasping intention recognition exceeds 89%on the training objects. When this method is extendedlyapplied to objects not included in the training set, the averageaccuracy exceeds 85%. The grasping experiment in the actualenvironment verifies the effectiveness of the proposed method.

I. INTRODUCTION

Grasping is one of the important actions in people’s dailylife, but the limbs of some disabled people cannot implementthe grasping function because of paralysis, amputation, etc.Wearable robots including exoskeletons [1], prosthetics [2],and supernumerary robotic fingers [3]–[5] can serve asalternative actuators.

Recognition of grasping intention is the premise for con-trolling wearable robots. The current main methods includeEMG-based methods [6], [7], EEG-based methods [8], [9],and action-based methods [10], [11]. Among these methods,the EMG-based method is not suitable for patients withhemiplegic paralysis, because the EMG signals of affectedlimbs have been distorted, from which it is difficult toidentify the intention of patients. The accuracy and dis-criminability of EEG signals obtained by the noninvasive

This work was supported by the National Natural Science Foundation ofChina under Grant (No. U1913207) and the International Science and Tech-nology Cooperation Program of China under Grant (No. 2017YFE0128300).(Corresponding author: Jian Huang)

Bo Yang, Jian Huang, Xiaolong Li and Xinxing Chen are with theKey Laboratory of Image Processing and Intelligent Control, School ofArtificial Intelligence and Automation, Huazhong University of Scienceand Technology, Wuhan 430074, China (e-mail:aleksibob@hust.edu.cn;huang jan@mail.hust.edu.cn; lixiaolong@hust.edu.cn; cxx@hust.edu.cn)

Caihua Xiong is with the School of Mechanical Science and Engineeringand the State Key Laboratory of Digital Manufacturing Equipment andTechnology, Huazhong University of Science and Technology, Wuhan430074, China (e-mail: chxiong@hust.edu.cn)

Yasuhisa Hasegawa is with Department of Micro-Nano Mechanical Se-cience and Engineering, Nagoya University, Furo-cho, Chikusa-ku, Nagoya464-8603, Japan (email: hasegawa@mein.nagoya-u.ac.jp)

brain-machine interface are relatively low, and it is easyto be interfered by external factors. Although the invasiveinterface can obtain accurate EEG signals, most patientscannot accept craniotomy. The action-based methods usesensors such as airbags and IMUs to obtain the action statesof the user’s limb to infer the user’s intention, but thesemethods are impossible for those patients who have lost theirlocomotivity.

Eye-tracking technology can be the feasible means ofgrasping intention recognition for such patients. The use ofeye movement as a control interface relies on the fact that ourgaze is proactive and directly correlated to action intentionsand cognitions [12]. Furthermore, the oculomotor systemcontrol function is usually retained even in the most severecases of paralysis related to muscular dystrophies or brainstroke. And in neurodegenerative diseases that affect the mo-tor system (such as stroke, Parkinson’s disease, or MultipleSclerosis), it is easier for patients to control their gaze pointrather than conducting stable skeletal movements [13].

Most of the studies using eye movement technology onlyfocus on reaching tasks, e.g., Roni et al. used gaze controllingrobotic arms to assist human hands in reaching tasks [14] ,and Shaft et al. used eye movement controlling robotic armsto perform tasks [15]. Other studies have adopted unnaturalmethods to recognize the grasping intention of patients, suchas asking users to deliberately blink or prolong the gaze,which will increase the cognitive load on users. It is not easyfor patients with nerve damage to complete these unnaturalactions.

In these case, natural human-robot interaction can beused to recognize the patients’ intention. Natural human-robot interaction uses natural action to obtain the user’sintention [16], [17], without imposing extra cognitive loadon the users. This is similar to the user’s control of theirown limbs for activities without long-term training.

Previous studies have shown that there are differences innatural gaze between grasping and viewing behaviours [18].However, these studies focus on the effects of different ob-jects on the first two fixations. To the best of our knowledge,there are only qualitative summaries of differences fromexperimental observations but no quantitative features thatcan be used for grasping intention recognition reported inexisting literatures.

On this basis, we study the relationships of the fixationsin the grasping tasks and viewing tasks when the thumband index finger are visible, which is a common situationwhile humans are conducting grasping tasks. We analyze

arX

iv:2

012.

0870

3v1

[cs

.RO

] 1

6 D

ec 2

020

these relationships and extract some quantitative featuresto identify gasping intention. Using the obtained features,we recognize the user’s grasping intention and verify theeffectiveness of the method. Our main contributions are asfollows:

1) We discover the relationships between the fixationselves, the fixations and objects, the fixations andgrasping points in the grasping tasks and the viewingtasks. Compared with the viewing tasks, the fixationsof the user are more centralized and tend to be closeto the grasp point of the index finger in the graspingtask.

2) We propose four features from the relationships forgrasping intention recognition. Then we compare theaccuracy of different features in recognition of grasp-ing intention. On this basis, we implement a naturalmethod to recognize the grasping intention by eyemovement.

3) We carry out grasping experiments with humanoidrobotic arms in actual daily scenes, which verifies theeffectiveness of the proposed method.

II. METHODS

A. System overview and architecture

We aim to create a system to recognize grasping intentionbased on human natural gaze data. The system consists ofa binocular eye-tracker, a scene camera, a head stand and ablack background (see Fig. 2). The block diagram in Fig. 1depicts an overview of our system. Individual modalities aredescribed in detail in the following.

1) Binocular eye-tracker: For 2D eye tracking, we usethe Pupil Core [19]. The eye-tracker is built from twoinfrared (IR) cameras used to photograph the eyes, yieldinga resolution of 192 × 192 pixels at a maximum frame-rateof 120 Hz. It has two extendable mounts for the infraredcameras, which can be adjusted to focus the IR camerason each eye respectively for individuals. There are two IRLEDs attached to the cameras to illuminate the pupil of theuser. Using the obtained infrared images, we can select theappropriate eye model and apply ellipse fitting parametersduring system operation to detect the pupil of the user, andthen use this information to estimate the user’s fixation point.

An Intel Realsense D435 RGB-D camera (Intel Corpora-tion, Santa Clara, California, USA) is mounted on the top ofthe eye-tracker as a scene camera, with a 3D printed frame.This camera is used to capture the user’s ego-centric scene.

2) Object centroid detection: In the experiment environ-ment, we use the OpenCV image processing library [20]to process live scene video streams and detect the object’scentroid in each frame with a contour-based algorithm. Toavoid the interference from ambient light, we use a solidcolor background. Besides, we filter the centroid of somedisturbing objects based on the area of the object contour.Because it is impossible to detect the real centroid of 3Dobjects in RGB images, we use the 2D centroid of objectsin images instead.

In real environment applications, we use Mask R-CNN [21] to quickly and accurately obtain object informa-tion. This machine learning-based method can effectively im-prove the robot system’s ability to perceive the environmentand identify objects that patients need to grasp in their dailylives.

3) Grasp position detection: In grasping tasks the thumband the other fingers play different roles. The thumb guidesthe hand straight to the object’s location, and then the otherfingers represented by the index finger grip around the objectto ensure a safe grasp [22]. We record the positions ofthe participants’ digits to obtain the relationship betweenthe fixation position and the grasping position. The grasp-ing positions are obtained through pre-experiment. In thepreliminary experiment, we ask participants to grasp eachobject in repeated fashion and use the average positions asthe grasping positions of the thumb and index finger.

B. Experiment

There are two categories of gaze behaviour: fixationsand saccades. Fixations are those times when our eyesessentially stop scanning about the scene, holding the centralfoveal vision in place so that the visual system can take indetailed information about what is being looked at [23]. Themain goal of the experiment is to investigate the differenceof fixation modes of the participants under different taskconditions of grasping and viewing.

Eight different participants carry out both viewing andgrasping tasks. Their ages range between 20 and 30. Prior tothe experiments, a short introduction is provided to these sub-jects, including technologies involved, the system setup andthe research purpose. During the experiment, a head standis used to fix the subjects’ head. In practical applications, ahead tracking module can be added to track the movementof the use’s head, which is not covered in this paper.

1) Procedure: The shapes that the participants are askedto grasp or view are magnetically attached to a black back-ground, which is placed in front of an iron plate (Fig. 2).We use three different white plastic objects for training,depicted in Fig. 2(b): a square, a cross, a T-shape (presentedin four different orientations). They are 10 mm thick. Thereare two types of grasping tasks, horizontal grasping andvertical grasping. The vertical T-shape is used for verticalgrasping tasks, while the horizontal T-shape is used forhorizontal grasping tasks. The square and cross shapes areused in both grasping tasks. For each type of the graspingtask, participants perform 4 shapes * 5 repetitions = 20practice trials, presented in random order. For the viewingtask, participants perform the same number of trials. We usetwo other shapes: a triangle and a bar (presented in 2 differentorientations) for testing.

The eye-tracker needs to be calibrated before the exper-iment starts. Participants rest their head on a head stand,50 cm in front of the monitor, which is the same distanceas in the experiment. Both eyes are calibrated using anine point calibration/validation procedure on the computermonitor. After the participants have performed half trials,

Fig. 1. The block diagram of our system. The user’s fixation positions are obtained by the eye-tracker, the object information is obtained by the imageprocessing algorithms, and the grasping position is obtained by pre-experiment. The features proposed in this paper are extracted from these data and usedto train the intent recognition model. In the task of intent recognition, the humanoid robot arm is controlled to complete the grasping action according tothe model recognition result.

(a) (b)

Fig. 2. The experimental scene and target shapes. (a) Schematic depiction of the setup and the way of grasping in the experiment. (b) Different shapesin the experiment.

the calibration is repeated. At the start of each trial, theparticipants fix at the start point and closes their eyes afterfixation.

For grasping tasks, participants start a trial with the eyesclosed and the hands resting on the lap. When hearing a“grasp” command, they open their eyes and immediately fixtheir eyes on the start point. Participants then stretch outtheir thumb and index finger to grasp the shape (Fig. 2(a)).Different grasping tasks have different grasping modes. Theuser’s fingers of the horizontal grasping tasks contact withthe left and right edges of the object, while the fingers ofthe vertical grasping tasks contact with the top and bottomedges. The trial is ended when the “finish” command isheard. After completing the trial, the participants return tothe start point with their hands on the lap and their eyesclosed, until hearing the verbal command to start next trial.

The procedure of the viewing task is similar to thegrasping task, with the exception that instead of being asked

to grasp, participants are asked to “view the shape” untilhearing a ”finish” command. The viewing process lasts forthree seconds. The duration is measured by the participantsthrough the pre-experiment. In the experiment, participantsperform 10 grasping tests repeatedly and we select themaximum time of task execution as the benchmark time inthe experiment.

III. DATA ANALYSIS

Among 640 grasping and viewing trails, the data of16 grasping trails (5%) and 24 viewing trails (7.5%) arerejected due to insufficient quality of eye data. We use thefixation detection algorithm to distinguish fixations from fastsaccades. The parameters of the algorithm recommended bythe manufacturer are set as follows:

d psmax= 3.01, durmin= 80ms, durmax = 400ms (1)

d psmax indicates the threshold of the distance between allgaze locations during a fixation. durmin indicates the mini-

mum duration in which the distance between every two gazelocations must not exceed d psmax and durmax indicates themaximum duration. After processing the gaze information,the two-dimensional coordinates of the fixation points ofparticipants in the trial are obtained. The coordinates arethe relative positions of the fixations to the centroid of theshape.

We compare the result of the grasping and viewing tasksthrough repeated measures analysis of variance (ANOVAs),taking the objects as within-subjects variable and the tasksas between-subjects variable. For all statistical tests, we use0.05 as the level of significance.

A. Number of fixation

Table II shows the average fixation times in one trial.On average, there are 8.18 (STD=0.99) fixation points foreach grasping task and 8.62 (STD=0.76) fixations for eachviewing task. The results of repeated measures ANOVA showthat the task type affects the number of fixations (F=270.60,p<0.001). But in one trial, the number of fixations will onlybe an integer, which makes it difficult to distinguish betweengrasping and viewing by the number of fixations.

B. Fixations locations

Fig. 3 illustrates the distribution of the fixation locationby showing the fixations in certain trials in the graspingand viewing tasks respectively. The centroid of the shapesis represented by the blue point. We can find that fixationshave different characteristics under different tasks. To effec-tively distinguish between grasping and viewing, we proposesome features and quantitatively analyze their relationship indifferent tasks.

In one trial, the user’s fixations are expressed as:

F(ti) = ( fx(ti), fy(ti)) (2)

i represents the number of fixation, and ti represents the timecorresponding to fixation i. The two-dimensional centroidcoordinates of the object are expressed as follows:

C(t) = (cx(t),cy(t)) (3)

where t represents the time of coordinates. The coordinatesof the user’s thumb and index finger grasp points in theegocentric video are expressed as:

G1(t) = (g1x(t),g1y(t))

G2(t) = (g2x(t),g2y(t))(4)

1) The average distance from the fixations to the centroid:The first feature we propose is the average distance from theuser’s fixation points to the centroid of the shape (ADF2C).This feature is used to describe the relationship between theparticipants’ fixations and the object:

ADF2C =1n

n

∑i=1

√( fx(ti)− cx(ti))

2 +( fy(ti)− cy(ti)2 (5)

If the detected centroid does not belong to the user’s targetobject, this feature will appear abnormal (too large), which

Fig. 3. The fixation position of participants on four shapes in certain trials.The centroid of the shapes is represented by the blue point.

means that the user’s target object can be determined by thisfeature.

Fig. 4 shows the distance between the fixations and thecentroid of the shape in grasping and viewing, includingthe mean value of the distances. Obviously, the curves ofADF2Cs intertwine in the grasping and viewing tasks, andtheir average values are so close that there is no significantdifference. A repeated measures ANOVA on the ADF2C withthe tasks as variable indicates that the ADF2C does not differbetween grasping and viewing (F=3.20, p = 0.07).

Fig. 4. The ADF2Cs in all trials. The solid line represents the result ofeach trial, and the dotted line represents the average of all results.

2) The average distance from the fixations to the graspingpoint: We find that there are also different relationshipsbetween the fixations and the grasp position in different tasks.The grasping positions of the participants are obtained fromthe pre-experiment. We consider using the average distancebetween the fixations and the grasping positions in eachtrial to reflect these relationships. These two features are

TABLE ISIGNIFICANT RESULTS FOR TWO KINDS OF GRASPING TASKS AND VIEWING TASKS REPEATED MEASURES ANOVAS ON THE ADF2C, THE ADF2I,

THE ADF2T, AND THE VAR

ADF2C ADF2I ADF2T VARVertical grasping and viewing F=0.046, p = 0.829 F=142.699, p <0.001 F=5.699, p=0.0176 F=284.699, p <0.001

Horizontal grasping and viewing F=3.781, p = 0.053 F=192.511, p <0.001 F=164.964, p <0.001 F=171.128, p <0.001

Grasp and viewing F=3.209, p = 0.074 F=270.596, p <0.001 F=80.568, p <0.001 F=437.500, p <0.001

TABLE IIFIXATIONS IN DIFFERENT TRIALS

Shape square cross cross down cross up cross right cross left mean

Viewing 8.71±0.83 8.35±0.70 8.60±0.89 8.57±0.70 8.08±0.87 8.62±0.66 8.62±0.80

Grasping 8.28±1.09 8.11±0.69 8.64±1.11 8.23±0.70 8.08±0.36 8.08±0.36 8.18±0.86

the average distance from the fixations to the index finger(ADF2I) and the average distance from the fixations to thethumb (ADF2T):

ADF2T =1n

n

∑i=1

√( fx(ti)−g1x(ti))

2 +( fy(ti)−g1y(ti)2

ADF2I =1n

n

∑i=1

√( fx(ti)−g2x(ti))

2 +( fy(ti)−g2y(ti)2

(6)

Fig. 5 shows the relationship between ADF2I and ADF2Tin all trials, respectively. We can find that ADF2I andADF2T are obviously different in grasping and viewingtasks. Furthermore, the mean value of ADF2I is significantlylower than that of ADF2T in grasping tasks. The reasonis that in the grasping tasks, the participants’ eyes focuson the index finger more than the thumb, which may helpstabilize the grasps. A repeated measures ANOVA on theADF2I shows a significant effect of the tasks (F=270.60,p<0.001). Similar to ADF2I, ADF2T is also affected by thetask(F=80.57, p<0.001). Table I reveals that this significanteffect is more obvious in horizontal grasping tasks than invertical grasping tasks (the value of F indicates the extentof the effect).

3) The relationship between fixation points: We also findthat in one trial, there is a relationship between the fixations.We first propose the variance (VAR) of the distances from allthe fixation points to the center as a quantitative descriptionof concentration:

O = (ox,oy) = (1n

n

∑i=1

fx(ti),1n

n

∑i=1

fy(ti))

di =

√( fx(ti)−ox)

2 +( fy(ti)−oy)2

M =1n

n

∑i=1

di

VAR =

n∑

i=1(di −M)2

n

(7)

It can be seen from Fig. 6 that the VARs in the graspingtasks are significantly smaller than those in the view tasks.

Fig. 5. The average distance from the fixation to the grasping point. Thesolid line represents the result of each trial, and the dotted line representsthe average of all results.

In the grasping tasks, the mean value of the VARs is 29.85.(STD=44.08), which is much smaller than the viewing tasks(MEAN=256.67, STD=181.05). This can be explained by thefact that in the grasping tasks, the participant’s attention andsight need to be more concentrated to complete the grasp,and the participant’s sight changes randomly across the targetduring the viewing task. A repeated measures ANOVA onthe VARs show a significant effect of the tasks (F=437.50,p<0.001). This effect is similar in horizontal grasping andvertical grasping.

Table I shows all significant results of the repeated mea-sures ANOVA on the ADF2C, the ADF2I, the ADF2T, andthe VAR.

IV. GRASPING INTENTION RECOGNITION

From the above data analysis, we can know that inthe viewing and grasping tasks, participants’ fixations havedifferent features, which show the potential of accuratelyestimating the participants’ grasping intention. Thanks to the

Fig. 6. The VARs in all trials. The solid line represents the result of eachtrial, and the dotted line represents the average of all results.

use of human natural eye behavior, people’s cognitive load isreduced and can be quickly accepted by users. This methodis especially suitable for patients with upper limb injuries.They can use gaze data to control and express their graspingintention, instead of relying on residual EMG or movement.

To evaluate the effect of different features on graspingintention recognition, we selected five feature combinations.

1) Combination1:ADF2T+VAR2) Combination2:ADF2I+VAR3) Combination3:ADF2C+ADF2T+VAR4) Combination4:ADF2C+ADF2I+VAR5) Combination5:ADF2C+ADF2I+ADF2T+VARSeveral different machine learning methods are used, in-

cluding support vector machine (SVM), K-Nearest Neighbor(KNN), stochastic gradient descent (SGD), and decision tree(DT). The total samples are randomly partitioned into 5equal-sized subsets. Among the 5 subsets, four subsets areused as the training set while the remaining one is used asthe testing set (Test1). For each methods,we use the 5-foldcross-validation to evaluate the accuracy of recognition. Theother test set (Test2) is collected from three other shapes thatdoes not exist in the training set, each of which contains 20samples.

From Fig. 7 we can learn that for different feature com-binations, Combination3 has the highest average accuracyon Test1, which is 89.7%±1.9%. The average recognitionaccuracy of the other four feature combinations on Test1also exceeds 88.8%. On Test2, Combination2 has the highestaverage recognition accuracy of 90.6%±4.5%. The averagerecognition accuracy of other feature combinations is alsoacceptable, the lowest of which is higher than 87%.

For different classifiers, the KNN has the highest averageaccuracy on Test1, which is 91.6%±0.3%. The SGD has thelowest grasping intention recognition accuracy on Test1, withan average of 85.1%±2.5%. But this is also an acceptableresult, which means that the influence of classifiers is small.

On Test2, the SVM (linear) achieves the best accuracy of

93.6%±0.3%. The accuracy of other classifiers also exceeds82%. The reason why the SVM classifier has higher accuracyon Test2 than Test1 may be that Test1 has a longer experi-mental process. This makes the participants become fatiguedand lead to abnormal fixation mode.

From the analysis, we can know that the features we dis-covered can effectively identify the user’s grasping intentionand can achieve excellent results with different classifiers.Furthermore, we have successfully applied the proposedfeature combinations to the newly appeared objects in thetest set to identify the user’s grasping intention. These resultsshow that the features we extracted from the gaze dataare robust and can reflect human intention, which can beeffectively applied to the recognition of users’ graspingintention.

V. GRASPING EXPERIMENT IN REAL ENVIRONMENT

We have carried out the application experiment in theactual daily scene with the humanoid arm. The experimentalscene is shown in the Fig. 8. We chose a cup that iscommonly used in daily life as the intended target. Weused the Mask R-CNN [21] object detection algorithm toextract the target information we need from the real andcomplex environment. Through the selected algorithm, wecan detect the mask of the cup from the user’s egocentricscene and use it to calculate the two-dimensional centroidof the object. We set up a sliding window with a durationof three seconds to sample the data. Then, the features areextracted by combining the centroid of the object, the grasp-ing points with the user’s fixation information. At the sametime, we use the Combination4 and the KNN as featuresand the classifier for grasping intention recognition. Theexperimental results show that, combined with the currentadvanced image processing algorithms, our proposed methodcan effectively identify the user’s grasping intention and suc-cessfully implement the grasp. The complete experimentalprocess is shown in the video in the material.

Four subjects participated in our experiment. To completethe objective evaluation, the subjects were asked to completethe Quebec User Evaluation of Satisfaction with assistiveTechnology (QUEST) questionnaire [24]. The results areshown in Table III. From the QUEST questionnaire we findthat although there is general agreement with effectiveness,easiness to use, and safety, there is displeasure on comfort.This is probably because the scene camera is relatively heavyand the eye-tracker is not a mature wearable product.

TABLE IIIQUEST EVALUATION RESULTS FOR THE HUMANOID ROBOTIC ARM

ASSIST SYSTEM

Safety Easyto Use Comfort Effec

-tiveNot satified at all 0 0 0 0Not very satisfied 0 0 1 0

More or less satisfied 1 0 1 0Quite satisfied 3 3 2 2Very satisfied 0 1 0 2

(a) (b)

Fig. 7. Classification accuracy of each classifier and combinations for grasping intention recognition. We perform 100 repeated experiments and used theaverage accuracy and standard deviation. (a) The accuracy of Test1. (b) The accuracy of Test2

Fig. 8. Application experiment in the real environment. The systemrecognizes the user’s grasping intention and controls the humanoid roboticarm to complete the grasp.

VI. DISCUSSION AND CONCLUSION

Recognition of grasping intentions is the premise forwearable robots to assist users in grasping, but in some cases,the eye movements are the only intention related activitiesthat can be observed. So we study the eye movements relatedto grasping.

We obtain the fixation messages of participants in graspingand viewing tasks through experiments. These messages arecombined with the target centroid and the grasping positionsto study the fixation laws under different tasks. Throughthe experimental results and statistical analysis, we find thatfixations are significantly different in the two tasks. In thegrasping task, the participants’ fixations are closer to thegrasp position of the index finger and concentrate on a smallarea. While in viewing tasks, the distribution of participants’fixations are arbitrary, which may be distributed in multipleparts of the target. This phenomenon has inspired us to usethese natural gaze fixations to identify the grasping intentionof participants.

We first propose four features- the ADF2C, the ADF2I, theADF2T, and the VAR to describe these differences quantita-tively. Utilizing the quantitative features, we can use machinelearning methods to recognize the user’s grasp intention,and further control the robot to help the user complete thegrasp. It is crucial that only the natural gaze fixations of thehuman are used in our method, and no additional actions orcommands are required. Thus, this method is very suitablefor patients with hemiplegia. However, not all users’ fixationsfollow the rules we have found. Some users’ fixations areclose the centroid or the grasp point of the thumb ratherthan the grasp point of the index finger when grasping.When the user knows the pose and shape of the object inadvance, he/she may be able to grasp it without staring at it.Additionally, the work of this paper explores the relationshipsof the users’ fixations in different tasks when the thumband index finger are visible. For some grasping tasks, thecontact position of the index finger is not visible. In the nextwork, we will explore the laws of fixations in grasping andviewing tasks when not all the grasping points are visible.Furthermore, we intend to apply the method to patients withhemiplegia so that they can complete daily grasping tasks.

In conclusion, we are exploiting the clues behind eyemovements, which are highly correlated to our motor actions.They are key factors in human motor planning and imply ourmovement intention. Consequently, gaze tracking becomesa rich source of information about user intent that canbe utilized. Fixations reflect the focus of human gaze. Wehave proved that only a few fixations are needed to quicklyand accurately recognize the user’s grasping intention. Thistechnology bears a tremendous potential as an interfacefor human-robot interaction. Moreover, eye-movements areretained in the majority of motor disabilities, which enhancesthe applicability of eye-tracking technology in the field ofhuman-robot interfaces for the paralysed and in prosthetics.Unlike EMG based and EEG based methods that require

long-term training, our method only needs natural eye move-ment and does not need training. The intention recognitionperformance of our system are subject to the accuracy ofgaze estimation. We hope to improve the accuracy of gazeestimation in the future to achieve more accurate recognitionresults. Our system allows for straightforward control of therobot for grasping. The application in the real environmenthas proved its effectiveness. Furthermore, the user’s fixationscan also provide the location information of the target. If wecombine the user’s intention with the position information ofthe object, the user can control the humanoid robotic arm tograsp the object with his eyes only. The ability to recognizethe user’s grasp intention from fixations enables novel waysof achieving the embodiment in robotic solutions that restoreor augment human functions.

REFERENCES

[1] A. M. Dollar and H. Herr, “Lower extremity exoskeletons and ac-tive orthoses: Challenges and state-of-the-art,” IEEE Transactions onrobotics, vol. 24, no. 1, pp. 144–158, 2008.

[2] J. T. Belter, J. L. Segil, A. M. Dollar, and R. F. Weir, “Mechanicaldesign and performance specifications of anthropomorphic prosthetichands: A review.” Journal of Rehabilitation Research & Development,vol. 50, no. 5, pp. 599–618, 2013.

[3] F. Y. Wu and H. H. Asada, “”hold-and-manipulate” with a single handbeing assisted by wearable extra fingers,” in 2015 IEEE InternationalConference on Robotics and Automation (ICRA). IEEE, 2015, pp.6205–6212.

[4] I. Hussain, G. Salvietti, G. Spagnoletti, and D. Prattichizzo, “Thesoft-sixthfinger: a wearable emg controlled robotic extra-finger forgrasp compensation in chronic stroke patients,” IEEE Robotics andAutomation Letters, vol. 1, no. 2, pp. 1000–1006, 2016.

[5] A. S. Ciullo, F. Felici, M. G. Catalano, G. Grioli, A. Ajoudani,and A. Bicchi, “Analytical and experimental analysis for positionoptimization of a grasp assistance supernumerary robotic hand,” IEEERobotics and Automation Letters, vol. 3, no. 4, pp. 4305–4312, 2018.

[6] G. Salvietti, I. Hussain, D. Cioncoloni, S. Taddei, S. Rossi, andD. Prattichizzo, “Compensating hand function in chronic stroke pa-tients through the robotic sixth finger,” IEEE Transactions on NeuralSystems and Rehabilitation Engineering, vol. 25, no. 2, pp. 142–150,2016.

[7] G. Marco, B. Alberto, and V. Taian, “Surface emg and muscle fatigue:multi-channel approaches to the study of myoelectric manifestationsof muscle fatigue,” Physiological measurement, vol. 38, no. 5, p. R27,2017.

[8] C. I. Penaloza and S. Nishio, “Bmi control of a third arm formultitasking,” Science Robotics, vol. 3, no. 20, p. eaat1228, 2018.

[9] J. E. Downey, N. Schwed, S. M. Chase, A. B. Schwartz, and J. L.Collinger, “Intracortical recording stability in human brain–computerinterface users,” Journal of neural engineering, vol. 15, no. 4, p.046016, 2018.

[10] J. Huang, W. Huo, W. Xu, S. Mohammed, and Y. Amirat, “Controlof upper-limb power-assist exoskeleton using a human-robot interfacebased on motion intention recognition,” IEEE Transactions on Au-tomation Science and Engineering, vol. 12, no. 4, pp. 1257–1270,2015.

[11] R. De Souza, S. El-Khoury, J. Santos-Victor, and A. Billard, “Recog-nizing the grasp intention from human demonstration,” Robotics andAutonomous Systems, vol. 74, pp. 108–121, 2015.

[12] M. Hayhoe and D. Ballard, “Eye movements in natural behavior,”Trends in cognitive sciences, vol. 9, no. 4, pp. 188–194, 2005.

[13] P. Cipresso, P. Meriggi, L. Carelli, F. Solca, D. Meazzi, B. Poletti,D. Lule, A. C. Ludolph, G. Riva, and V. Silani, “The combined useof brain computer interface and eye-tracking technology for cognitiveassessment in amyotrophic lateral sclerosis,” in 2011 5th InternationalConference on Pervasive Computing Technologies for Healthcare(PervasiveHealth) and Workshops. IEEE, 2011, pp. 320–324.

[14] R. O. Maimon-Mor, J. Fernandez-Quesada, G. A. Zito, C. Konnaris,S. Dziemian, and A. A. Faisal, “Towards free 3d end-point controlfor robotic-assisted human reaching using binocular eye tracking,” in2017 International Conference on Rehabilitation Robotics (ICORR).IEEE, 2017, pp. 1049–1054.

[15] A. Shafti, P. Orlov, and A. A. Faisal, “Gaze-based, context-awarerobotic system for assisted reaching and grasping,” in 2019 Interna-tional Conference on Robotics and Automation (ICRA). IEEE, 2019,pp. 863–869.

[16] F. Ferland, D. Letourneau, A. Aumont, J. Fremy, M.-A. Legault,M. Lauria, and F. Michaud, “Natural interaction design of a humanoidrobot,” Journal of Human-Robot Interaction, vol. 1, no. 2, pp. 118–134, 2013.

[17] A. Valli, “The design of natural interaction,” Multimedia Tools andApplications, vol. 38, no. 3, pp. 295–305, 2008.

[18] A.-M. Brouwer, V. H. Franz, and K. R. Gegenfurtner, “Differences infixations between grasping and viewing objects,” Journal of Vision,vol. 9, no. 1, pp. 18–18, 2009.

[19] M. Kassner, W. Patera, and A. Bulling, “Pupil: an open source platformfor pervasive eye tracking and mobile gaze-based interaction,” inProceedings of the 2014 ACM international joint conference onpervasive and ubiquitous computing: Adjunct publication, 2014, pp.1151–1160.

[20] A. Kaehler and G. Bradski, Learning OpenCV 3: computer vision inC++ with the OpenCV library. ” O’Reilly Media, Inc.”, 2016.

[21] K. He, G. Gkioxari, P. Dollar, and R. Girshick, “Mask r-cnn,” inProceedings of the IEEE international conference on computer vision,2017, pp. 2961–2969.

[22] P. Haggard and A. Wing, “On the hand transport component ofprehensile movements,” Journal of motor behavior, vol. 29, no. 3,pp. 282–287, 1997.

[23] K. Rayner, “The 35th sir frederick bartlett lecture: Eye movements andattention in reading, scene perception, and visual search,” Quarterlyjournal of experimental psychology, vol. 62, no. 8, pp. 1457–1506,2009.

[24] L. Demers, R. Weiss-Lambrou, and B. Ska, “The quebec user evalua-tion of satisfaction with assistive technology (quest 2.0): an overviewand recent progress,” Technology and Disability, vol. 14, no. 3, pp.101–105, 2002.