8

Click here to load reader

Haptic Feedback Enhances Grip Force Control of sEMG-Controlled Prosthetic Hands in Targeted Reinnervation Amputees

  • Upload
    j-e

  • View
    214

  • Download
    1

Embed Size (px)

Citation preview

Page 1: Haptic Feedback Enhances Grip Force Control of sEMG-Controlled Prosthetic Hands in Targeted Reinnervation Amputees

798 IEEE TRANSACTIONS ON NEURAL SYSTEMS AND REHABILITATION ENGINEERING, VOL. 20, NO. 6, NOVEMBER 2012

Haptic Feedback Enhances Grip Force Control ofsEMG-Controlled Prosthetic Hands in Targeted

Reinnervation AmputeesKeehoon Kim and J. Edward Colgate

Abstract—In this study, we hypothesized that haptic feedbackwould enhance grip force control of surface electromyography(sEMG)-controlled prosthetic hands for targeted reinnervation(TR) amputees. A new miniature haptic device, a tactor, that candeliver touch, pressure, shear, and temperature sensation, allowsmodality-matching haptic feedback. TR surgery that createssensory regions on the patient’s skin that refer to the surface ofthe missing limb allows somatotopic-matching haptic feedback.This paper evaluates the hypothesis via an sEMG-controlledvirtual prosthetic arm operated by TR amputees under diversehaptic feedback conditions. The results indicate that the grip forcecontrol is significantly enhanced via the haptic feedback. However,the simultaneous display of two haptic channels (pressure andshear) does not enhance, but instead degrades, grip force control.

Index Terms—Grip force, mechanical haptic display, surfaceelectromyography (sEMG)-controlled prosthesis, sensory feed-back, targeted reinnervation (TR), upper extremity prosthesis.

I. INTRODUCTION

S ENSORY perception plays an important role in object ma-nipulation [1], [2]. Haptic information subserves a wide va-

riety of manual activities including stable grasping, activatingbuttons, knobs and other interface, detecting shape, complianceand texture, and so on [3]–[9]. Also, the sensorimotor systemmust use haptic information such as the grip force magnitudesand directions during in-hand manipulation [10]–[15]. Humanskin, especially the glabrous skin of the fingertips, is richly in-nervated by a variety of specialized mechanoreceptors and freenerve endings that provide haptic information involved in pre-cision grip control of unanticipated and slippery objects [1], [5],[16].All of this sensory perception information, however, is lost

when a limb is amputated. Thus, the problem of providing hapticfeedback to amputees has been approached in a number of dif-ferent ways. A straightforward approach is sensory substitu-tion [17]. For instance, vibration and electrocutaneous displays

Manuscript received July 19, 2011; revised November 22, 2011; acceptedJune 04, 2012. Date of publication July 26, 2012; date of current versionNovember 02, 2012.K. Kim is with the Interaction and Robotics Research Center, Korea Institute

of Science and Technology, Seoul 136-791, Korea (e-mail: [email protected]).J. E. Colgate is with the Department of Mechanical Engineering, North-

western University, Evanston, IL 60091 USA (e-mail: [email protected]).Color versions of one or more of the figures in this paper are available online

at http://ieeexplore.ieee.org.Digital Object Identifier 10.1109/TNSRE.2012.2206080

mounted on various parts of the body including the forearm,abdomen, and back have been used to represent tactile sensa-tion, proprioception, or the grip pressure of a prosthetic hand[18]–[20]. Although this method may be used to realize minia-ture and reliable haptic displays, it is by no means evident thatthe results are natural, intuitive, or desirable to amputees. Itseems natural that, in order to achieve intuitive haptic feed-back, it is beneficial to satisfy two conditions: 1) somatotopicmatching and 2) modality matching. For example, when wepress an object with the prosthetic thumb, if we feel the pres-sure at the thumb, then we can say that modality (pressure) andsomatotopic (thumb) matching conditions are satisfied.Modality-matched displays of grip pressure have also been

developed [21]–[24]. For instance, Meek et al. explored grippressure feedback with a myoelectrically controlled prostheticarm. They mounted a servo-controlled “pusher” to the socket.The pusher pressed into the skin an amount proportional to theforce in the terminal device, a method that the authors termed“extended physiologic taction” (EPT). It was reported that theability to grip a brittle object without breaking it was improvedwith EPT [21]. This method provides modality-matched dis-play of grip pressure, however, somatotopic match is not stillachieved.In order to provide a somatotopic match, our method was

tested with the subjects who had undergone targeted reinner-vation (TR) surgery. TR reroutes severed nerves from the am-putated limb to residual muscles and skin [25]–[28]. In thissurgery, both the efferent and the afferent nerves are reroutedto the residual muscles and the skin, respectively. Thus, whenthe TR patient thinks about moving his or her missing limb, ef-ferent nerve signals contract the residual muscles. Surface elec-tromyography (sEMG) signals from these muscles can be usedas command signals for a prosthetic arm. In effect, the musclesare both keeping the residual nerves healthy and serving as anamplifier of neural signals. Reinnervation also creates sensoryregions on the patient’s skin (for instance, the chest or upperarm) that are referred to the surface of the missing limb. As aresult, TR provides a practical means of somatotopic matching.Modality matching is also possible with TR patients. At least

some patients perceive a variety of stimuli in a normal manner:pressure applied to the reinnervated skin is perceived as pressureon the hand; vibration maps to vibration; hot/cold map to hot/cold; and sharp objects can be distinguished from dull [29]–[32].We developed a multi-functional haptic device, a tactor, that

can display touch, pressure, vibration, shear force, and temper-ature to the skin of an upper-extremity amputee, especially on

1534-4320/$31.00 © 2012 IEEE

Page 2: Haptic Feedback Enhances Grip Force Control of sEMG-Controlled Prosthetic Hands in Targeted Reinnervation Amputees

KIM AND COLGATE: HAPTIC FEEDBACK ENHANCES GRIP FORCE CONTROL OF SEMG-CONTROLLED PROSTHETIC HANDS 799

TR patients [33]. Thus, it should be possible to create an in-tuitive haptic-feedback system by placing sensors on the pros-thetic hand ([34], [35]) and using these to drive correspondinghaptic displays using a tactor (modality matching) on the TRskin (somatotopic matching).This study aims to verify our hypothesis that this type of

haptic display system is able to enhance control of grip forceduring amputees’ prosthesis operation. We use a grip-and-lifttask that calls for minimum pressure. The grip-and-lift tasksplay an important role in manipulating objects in activities ofdaily living (ADL) [1], [7], [36]. First, from an energy stand-point, the human must grip an object using minimum pressureso that fatigue can be minimized. Second, the controllability ofthe gripping force allows humans to manipulate breakable ob-jects such as eggs. In this sense, the control of grip force duringthe grip-and-lift task could be a measure to verify our hypoth-esis. In this study, we used a simplified grasp and manipulationtask, a two-finger grip-and-lift used extensively in previous lit-erature [1], [2], [5], [6], [37], [38].

II. MATERIALS AND METHOD

A. Subjects

Two amputees who had undergone TR surgery participatedin the experiments as subjects. Subject SD1, a 55-year-old malebilateral shoulder disarticulated amputee, had four of thesenerve-to-muscle transfers on his left side. The details of theTR surgery method are described in [25]–[28]. Subject SD2, a25-year-old female left unilateral short transhumeral (functionalshoulder disarticulation) amputee, had four nerve-to-muscletransfers. A detailed description can be found in [26].

B. Apparatus

1) sEMG Electrodes: In this test, four self-adhesive elec-trodes were placed at the sites of the patient’s TR chest skin,chosen previously through clinical evaluation, and corre-sponding to the desired phantom limb motions, i.e., elbowflexion, elbow extension, hand close, and hand open [26], [27],[29], [39].The sEMG signals were amplified and band-pass filtered

from 5 to 400 Hz and converted to the velocity commandsignals for the motion of the virtual prosthetic arm. In thevirtual environment, the patient was allowed to control thevirtual prosthetic arm to grip and lift a virtual object. Details ofthe virtual environment will be explained in Section II-B3.2) Tactor: In our experiments, we used a tactor that can pro-

vide multiple sensations such as touch, pressure, shear, and tem-perature [33]. A tactor is essentially a small and light two de-gree-of-freedom (DOF) robot [Fig. 4(b)]: one axis for the direc-tion normal to the skin and another for the directions tangentialto the skin. The tactor is capable of generating vibrations at leastan order of magnitude (20 dB) greater than the threshold of thefingertips [40] across a wide frequency range. The tactor canfollow tapping, pressure, and vibration commands in real time.It can display 9 N maximum force with 0.1 N resolution and 10mm stroke for both pressure and shear [33].In this study, when the patient manipulates the virtual object

with the virtual prosthetic arm, the grip force (pressure), and/or

Fig. 1. Shaded regions correspond to the perceived location of a stimulus on aphantom hand. (a) SD1. (b) SD2.

Fig. 2. Dynamics in the virtual environment.

the shear force are displayed by the tactor. The computationaldetails are given in Section II-B3.A tactor was placed on SD1’s TR skin at a location corre-

sponded to the palmar region of the phantom hand, as illustratedin Fig. 1(a). A tactor was placed on SD2’s TR skin at a locationcorresponded to the palmar region of the phantom hand, as il-lustrated in Fig. 1(b). SD2 described the pressure sensation asbeing spread out from the first contact point to the larger area,as illustrated in Fig. 1(b).3) Virtual Prosthesis and Environments: A real prosthetic

hand, of course, could have been used in this study, but it wasreplaced by a virtual prosthesis and virtual environment for thefollowing reasons. 1) We had only a limited time-slot with thepatients much of which would have been consumed by the fit-ting of a real prosthetic arm and the training session. 2) Thereis no adequate prosthetic arm and we wanted to isolate our re-search question from other issues such as prosthetic arm controland sensing. 3) In order to focus on our hypothesis, a simpleexperimental setup was deemed best. This will be discussed ingreater depth in Section IV. 4) In previous literature [41], [42],

Page 3: Haptic Feedback Enhances Grip Force Control of sEMG-Controlled Prosthetic Hands in Targeted Reinnervation Amputees

800 IEEE TRANSACTIONS ON NEURAL SYSTEMS AND REHABILITATION ENGINEERING, VOL. 20, NO. 6, NOVEMBER 2012

Fig. 3. Block diagram of the experiment setup comprising a patient, sEMG electrodes, a virtual prosthesis and environments, a visual display, and a tactor.

TABLE IPARAMETERS USED IN VIRTUAL ENVIRONMENT

virtual reality simulation was successfully shown to be suitablefor experiments in the absence of an adequate prosthetic hand.Fig. 2 illustrates the virtual environment consisting of a vir-

tual prosthetic arm, an object, and the ground. In the virtual en-vironment, the prosthetic arm and the virtual object are control-lable. The virtual prosthetic arm is controlled by the patient viasEMG signals from the TR muscle. The virtual object can bemanipulated by the virtual prosthetic arm. The sEMG signals areused to determine the desired velocity command for the virtualprosthetic arm, and . The -direction motion is as-sociated with the patient’s phantom hand grasping motion, handclose (negative -direction) and hand open (positive -direc-tion). The -direction motion is associated with elbow flexion(positive -direction) and elbow extension (negative -direc-tion). Therefore, the patient is able to grip the virtual block usingthe hand open/close motion and move it in -direction usingthe elbow flexion/extension motion subject to the following dy-namics. The virtual prosthetic hand and object were assumed tobehave as rigid bodies that have no geometric warping or dis-tortion, so that the subjects could not guess the grip force fromthe visual information. Thus, the environment has the followingdynamics:

(1)

(2)

(3)

(4)

where is the position of the object, is gravitational acceler-ation, is external force from the ground, and is the fric-tion force between the virtual prosthetic hand and object, isthe friction coefficient, is the pressure applied to the objectby the hand. , , are mass, stiffness, and damping coeffi-cient of the object, respectively. and represent theresultant position of the hand and the elbow motions. andare the width of the object and the radius of hand, respectively.Table I shows the parameters used in this study. The virtual en-

vironment is displayed to the subject by a video projector in realtime.

C. Interface

Fig. 3 summarizes the experimental setup including the sub-jects and apparatus mentioned above. The amplified and band-pass filtered sEMG signals were used as velocity commandsfor the motion of virtual prosthesis interacting with virtual ob-jects. The subjects could see the virtual prosthetic hand andthe virtual objects [Fig. 4(d)] and feel the pressure feedbackunder haptic feedback conditions. The force and position ofthe tactor were controlled by MATLAB (Mathworks, Natick,MA) Simulink XPC toolbox through a data acquisition board(Sensoray 526). The tactor [Fig. 4(b)] was mounted on a mi-cromanipulator shown in Fig. 4(a) (Narishige MM-3) that wasattached to an adjustable armature (Leica Microsystems, Ban-nockburn, IL). The tactor pose was normal to the skin surface[Fig. 4(c)]. To prevent distraction during the tests, subjects worenoise cancellation headsets. Testing locations were shaved ifnecessary. All subjects were seated in an adjustable chair.

D. Protocol

In this study, the subjects were able to independently operatetwo degrees-of-freedom motion of a virtual hand using sEMGsignals from the TR muscle. They were allowed to manipu-late virtual objects using a grip-and-lift operation with the vir-tual prosthetic arm. The tactor displayed haptic feedback whenthe virtual hand interacted with the virtual object. Grip forceand task completion speed were used as task performance mea-sures. All procedures were performed with informed consentand approved by the Northwestern University Institutional Re-view Board. The experiments lasted 3 h. The subjects were al-lowed to take a rest at anytime during experiments if they felttired or uncomfortable.In the practice session prior to the main experiment, the sub-

ject was allowed to become familiar with the devices and pro-cedures. During the practice session, we adjusted the pressurerange, the stimulation point on the TR skin, and the amplifica-tion level of sEMG signals that determines speed of the virtualprosthesis to make the subject feel comfortable.In the main experiment, the subject was asked to grip the vir-

tual block with minimum pressure, to bring it up to the prede-fined target zone, and to release the block, allowing it to falldown. At the same time, the tactor delivered haptic sensationsto the subject. In order to encourage the subjects to concen-trate on using minimum pressure during trials, scores were dis-played just after each trial (the “grip-lift-release” task) was com-pleted. The score was calculated as: score

Page 4: Haptic Feedback Enhances Grip Force Control of sEMG-Controlled Prosthetic Hands in Targeted Reinnervation Amputees

KIM AND COLGATE: HAPTIC FEEDBACK ENHANCES GRIP FORCE CONTROL OF SEMG-CONTROLLED PROSTHETIC HANDS 801

Fig. 4. Experimental setup. (a) A micromanipulator (Narishige MM-3)attached to an adjustable armature (Leica Mirosystems, Bannockburn). (b) Thetactor integrated on the micromanipulator. (c) The tactor and a socket interfaceon a TR patient. (d) Graphical display including virtual environment shown tosubjects.

, where represents a breaking force,(9.0 N), and was the minimum force needed to lift theblock, (see Table I). We called the set of trialsduring which one haptic feedback condition was maintained, anexperimental set. The subjects repeated the grip-and-lift trialsuntil the accumulated score (initial score is 0) hit a maximum orminimum bound, i.e., 240 or , after which the next exper-imental set with a randomly predetermined alternative hapticfeedback condition began. A four-alternative haptic feedbackparadigm was used, i.e., “none” (N), “pressure” (P), “shear”

(S), and “pressure and shear” (PS). In order to prevent repeatingtoo many trials to complete an experimental set, 1) a trial wasassumed a failure and scored if the task completion timewas greater than 90 s, and 2) an experimental set was assumeda failure and the next experimental set began if the number oftrials was more than 10.

III. RESULTS

SD1 repeated 20 experiment sets with a randomly predeter-mined order for the four alternative haptic feedback conditions.An experiment set typically lasted 2 min and breaks were givenafter the sixth and tenth sets. Since SD2 was not comfortablewith the shear display in a practice session, SD2 repeated 14experiment sets for only two haptic feedback conditions the“none” (N) and “pressure” (P) conditions. An experiment settypically lasted 5 min and breaks were given after the fifth andninth experimental sets as the subject felt fatigue in their TRmuscles.

A. SD1

Fig. 5(a) shows the average grip pressure of the trials in eachexperimental set with the lower and upper quartile values andmedian indicated. Black, gray, yellow, and orange colored barsrepresent the “none,” “pressure,” “shear,” and “pressure andshear” feedback, respectively. As expected, no learning curvepatterns were observed because the subject had been trained suf-ficiently in the training session.The subject completed most tasks within 5 s except for a few

cases, as shown in Fig. 5(b). Note that there was no correla-tion between the task-completion speed and the grip pressure.In other words, spending more time did not lead to better gripforce control performance [see Fig. 5(a) and (b)]. For example,the subject spent longer time to complete the first, the fifth, andthe eleventh experimental set, however, it did not lead to higherperformance.Fig. 6 shows the average pressure and task completion time

under each haptic feedback condition. Fig. 6(a), the result of“none” condition was significantly higher than those of otherconditions. A balanced one-way ANOVA test was performed(ANOVA1.m, Statistics Toolbox, MATLAB, Mathworks). Thep-value of the result of SD1 was . Thus, we canconclude that haptic feedback can enhance grip force control.Fig. 6(b) shows that haptic feedback did not improve the taskcompletion speed. The results of the task-completion time wereconsistent with the results of [21].Interestingly, the results of the “pressure and shear” con-

dition showed inferior performance compared to the singularhaptic feedback conditions. The subject may have been con-fused by the simultaneous pressure and the shear information.Another possibility is that SD1’s reinnervated skin had a loweracuity for stretch than for pressure (independent measurementsof stretch sensitivity have not been made). Yet, another possi-bility is that simply displaying the resultant force on the TR skinvia a tactor was not sufficient to provide grip force information.The tactor simply played back the force and its direction on theTR skin indiscriminate of skin-stretch conditions. However, theskin-stretch is known as an important factor in delivery of slipinformation in grip-lift tasks [15], [43], [44]. Also, the method

Page 5: Haptic Feedback Enhances Grip Force Control of sEMG-Controlled Prosthetic Hands in Targeted Reinnervation Amputees

802 IEEE TRANSACTIONS ON NEURAL SYSTEMS AND REHABILITATION ENGINEERING, VOL. 20, NO. 6, NOVEMBER 2012

Fig. 5. Experimental results for SD1: none (black), pressure (gray), shear (yellow), pressure and shear (orange): (a) average of grip pressure in each experimentset with upper and lower quartile, (b) average of task completion time in each experiment set with upper and lower quartile.

Fig. 6. Averages of the haptic feedback condition for SD1: none (black), pres-sure (gray), shear (yellow), pressure and shear (orange). (a) Average of grippressure under each haptic feedback condition, p-value of one-way ANOVAtest, . The average grip pressure in the “none,” “pressure,” “shear,”and “pressure and shear” feedback conditions were 7.7 (7.0 at quartile,8.5 at quartile), 6.3(5.5, 7.0), 6.2(5.1, 6.9), and 6.8(5.7, 8.0), respectively.(b) Average task-completion time under each haptic feedback condition. Theaverage task completion time in the “none,” “pressure,” “shear,” and “pressureand shear” feedback conditions were 2.7 (2.6 at quartile, 6.9 at quar-tile), 2.9(2.5, 3.1), 2.9(2.4, 3.2), and 2.8(2.5, 4.4), respectively.

by which the tactor delivers stimulation to the TR skin, by in-dentation rather than light scanning, can negatively affect thehaptic perception.The “shear” condition shows a slightly better result than the

“pressure” condition. It seems as though the subject appeared tounderstand how to get a higher score by exploiting the physical

relationship between friction and pressure. In the “none” condi-tion, the subject required more time to assess slip, although thisdid not lead to a higher score.

B. SD2

Fig. 7(a) shows the average of grip pressure for trials in eachexperiment set. Black and gray colored bars indicate the “none”and “pressure” feedback conditions. No learning curve patternswere observed. The grip pressure under the “pressure” feedbackcondition is lower than that under “no haptic feedback” condi-tion. Under the “pressure” feedback condition, the subject usedpressure just above the minimum requirement for lifting (5.8 N)throughout the entire experiment.In Fig. 7(b), the subject completed most tasks within 15 s

except for a few cases. Note that there is no correlation betweenthe task-completion speed and the pressure records analogousto the results of SD1 [see Fig. 7(a) and (b)].Fig. 8 shows the average of pressure and task completion

time for each haptic feedback condition. In Fig. 8(a), the re-sult of “none” condition was significantly higher than for the“pressure” condition. The p-values of the result of SD2 was

. Thus, the grip pressure control was greatly enhancedunder the latter compared to the former. Again, Fig. 8(b) showsthat haptic feedback did not improve the task completion speed.

IV. DISCUSSION

In this study, we examined the hypothesis that haptic feed-back is able to enhance grip pressure control. We did so viatwo finger grip-lift task similar to previous studies of intactsubjects [1], [2], [5], [6], [37], [38]. From the results, wecan conclude that haptic feedback enhance grip force con-trol of sEMG-controlled prosthesis in targeted reinnervationamputees, but that delivery of multiple haptic sensations (si-multaneous pressure and shear) shows poorer performance than

Page 6: Haptic Feedback Enhances Grip Force Control of sEMG-Controlled Prosthetic Hands in Targeted Reinnervation Amputees

KIM AND COLGATE: HAPTIC FEEDBACK ENHANCES GRIP FORCE CONTROL OF SEMG-CONTROLLED PROSTHETIC HANDS 803

Fig. 7. Experimental results for SD2: none (black), pressure. (a) Average of grip pressure in each experiment set with upper and lower quartile. (b) Average oftask completion time in each experiment set with upper and lower quartile.

Fig. 8. Average of each haptic feedback conditions for SD2: none (black). pres-sure (a) Average of grip pressure in each haptic feedback conditions, p-value ofone-way ANOVA test, . The average pressure under the “none”and “pressure” feedback conditions were 8.0 (7.0 at quartile, 9.0 atquartile) and 7.3 (6.8–7.7), respectively. (b) Average of task completion time ineach haptic feedback conditions. The average task completion time under theno haptic feedback and pressure feedback conditions were 14.6 (11.5 atquartile, 16.1 at quartile), and 14.9 (11.9, 16.0) respectively.

single modality (either pressure or shear) feedback with ourspecific implementation.The results of this experiment must be interpreted in light

of multiple control deficits beyond impaired sensory feedback.Other deficits included the use of sEMG as a control signal,the inability of TR patients to contract muscles and sense touchsimultaneously, and subject fatigue.In this experiment, subjects’ intentions were based on sEMG

signals from the TR muscle. Analogous to rate control of pros-theses [41], [45], [46], sEMG was used to command velocityof the virtual gripper. While this mapping is not natural and cer-tainly makes it more difficult to control grip force, it nonethelessrepresents the state-of-the-art in clinical practice.

While TR surgery enables natural coordination of limb mo-tion and somatotopically matched feedback [25]–[28], there isanecdotal evidence that TR patients typically cannot feel tac-tile sensations during muscle contraction. Thus, in this experi-ment, the subjects employed a hybrid technique, contracting theTR muscle for motion and relaxing the muscles while sensingpressure. In this experiment, the task was relatively simple andthe gripping and lifting motions were decoupled. However, asthe task becomes more difficult (more than the coupled 2 de-grees-of-freedom), a different control strategy (or advances inthe TR surgery) will be needed.A final difficulty with the experiment was that TR patients

got tired after a number of repetitions of grip-and-lift and theirsEMG signals became weak and hard to decode. This was ad-dressed by giving subjects ample time for rest, but it underscoresthe point that minimization of grip pressure is an importantissue. If a patient had no force feedback, he or she used greaterlevels of muscle contraction (or the same levels for longer pe-riods of time) to ensure an adequate grip. Pressure feedback en-abled them to minimize muscle contraction.Another challenge that will need to be addressed in future

designs of the tactor is movement of the skin surface duringmuscle contraction. We have found that muscle contraction maycause the skin to displace by as much as 1 cm in the directionof normal to skin. At present, this is as much as the maximumdisplacement of the tactor. A related problem is that some TRtissues can be quite compliant, requiring more displacement toproduce highly perceptible forces. We were fortunately able tolocate the tactor above the less fatty, less compliant skin near theclavicle so that full tactor displacement was not needed in thisexperiment, but the issue remains for future work. One approachmay be to mount the tactor on the skin, and another approachis to preload the tactor enough that contact is never lost. Bothapproaches, however, present challenges with regard to com-fort. Indeed, both of our subjects indicated that they preferredno preloading.

Page 7: Haptic Feedback Enhances Grip Force Control of sEMG-Controlled Prosthetic Hands in Targeted Reinnervation Amputees

804 IEEE TRANSACTIONS ON NEURAL SYSTEMS AND REHABILITATION ENGINEERING, VOL. 20, NO. 6, NOVEMBER 2012

Fig. 9. Interview results.

There is a fundamental issue that pertains to tactile acuity.When humans manipulate objects, they use the glaburous skinof the hands that is much more richly innervated compared tohairy skin such as that found on the chest. It has recently been re-ported that “the Weber Ratio values for reinnervated skin werealways within the control group 95% confidence interval andclosely matched to contralateral chest skin,” [30] where the con-trol group was ten able bodied subjects and the TR subjects in[30] were the same with SD1 and SD2 in this article thoughthe stimulation points were different. Thus, TR subjects did nothave substantially different tactile acuity on their reinnervatedversus their contralateral intact side. Sensory resolution of TRskin is much poorer than that of the hand. This would appearto impose a fundamental limit on the performance that can beachieved in tasks such as grip force regulation.SD1 found ways to get a high score that we did not expect.

He used visual slippage as an indicator of minimum pressurein addition to the haptic feedback. In particular, he gripped theblock and pushed it downward against ground, then, he adjustedthe pressure at a level just higher than when the visual slippingmotion occurred. In addition, he modulated the pressure duringthe lifting up motion. As a result, Fig. 5(a) shows a grip pressureless than 5.8 N, the minimum required for static lifting (note thatthe pressure was recorded when the task was completed). In arecent study [45], the same subject was able to easily control2-DOF simultaneously in many different combinations. Thus,SD1’s simultaneous control of grip and lifting is not a surprisingresult.Finally, we asked the subjects for qualitative feedback

(Fig. 9). They agreed that the task was difficult with visualinformation only, and that but the tactor made it easier. Theyconfirmed that, during the experiments, they tried to use min-imum grip force as we intended. They did not strongly agreethat the tactor delivered a “realistic” haptic sensation. Thus,while the tactor provided sufficient information to control gripforce in a simple grip-and-lift task, it seems evident that moreadvanced techniques need to be developed for more realistichaptic sensation and dexterous manipulation.

ACKNOWLEDGMENT

The authors would like to thank T. A. Kuiken, L. A. Miller,P. D. Marasco, B. Lock, and M. Wu for their assistance andadvice.

REFERENCES[1] R. Johansson and G. Westling, “Roles of glabrous skin receptors and

sensorimotor memory in automatic control of precision grip whenlifting rougher or more slippery objects,” Exp. Brain Res., vol. 56, no.3, pp. 550–564, 1984.

[2] J. Monzee, Y. Lamarre, and A. M. Smith, “The effects of digital anes-thesia on force control using a precision grip,” J. Neurophysiol., vol.89, no. 2, pp. 672–683, 2003.

[3] J. R. Flanagan and A. M. Wing, “The stability of precision grip forcesduring cyclic arm movements with a hand-held load,” Exp. Brain Res.,vol. 105, no. 3, pp. 455–464, 1995.

[4] G. Cadoret and A. Smith, “Friction, not texture, dictates grip forcesused during object manipulation,” J. Neurophysiol., vol. 75, pp.1963–1969, 1996.

[5] P. Jenmalm and R. S. Johansson, “Visual and somatosensory informa-tion about object shape control manipulative fingertip forces,” J. Neu-rosci., vol. 17, no. 11, pp. 4486–4499, 1997.

[6] H. E. Wheat, L. M. Salo, and A. W. Goodwin, “Human ability to scaleand discriminate forces typical of those occurring during grasp and ma-nipulation,” J. Neurosci., vol. 24, no. 13, pp. 3394–3401, 2004.

[7] R. Reilmann, A. M. Gordon, and H. Henningsen, “Initiation and de-velopment of fingertip forces during whole-hand grasping,” Exp. BrainRes., vol. 140, no. 4, pp. 443–452, 2001.

[8] A. M. Smith, G. Gosselin, and B. Houde, “Deployment of fingertipforces in tactile exploration,” Exp. Brain Res., vol. 147, no. 2, pp.209–218, 2002.

[9] R. H. Lamotte, R. M. Friedman, C. Lu, P. S. Khalsa, and M. A. Srini-vasan, “Raised object on a planar surface stroked sensory signals neuralpopulations underlying tactile perception and manipulation,” J. Neuro-physiol., vol. 106, no. 1, pp. 2446–2466, 1998.

[10] J. C. Rothwell, M. M. Traub, B. L. Day, J. A. Obeso, P. K. Thomas, andC. D. Marsden, “Manual motor performance in a deafferented man,”Brain, vol. 105, no. 3, pp. 515–542, 1982.

[11] C. Hager-Ross, K. J. Cole, and R. S. Johansson, “Grip-force responsesto unanticipated object loading: load direction reveals body- and grav-ityreferenced intrinsic task variables,” Exp. Brain Res., vol. 110, no. 1,pp. 142–150, 2004.

[12] K. J. Cole, C. M. Steyers, and E. K. Graybill, “The effects of gradedcompression of the median nerve in the carpal canal on grip force,”Exp. Brain Res., vol. 148, no. 2, pp. 150–157, 2003.

[13] A. W. Goodwin, P. Jenmalm, and R. S. Johansson, “Control of gripforce when tilting objects: Effect of curvature of grasped surfacesand applied tangential torque,” J. Neurosci., vol. 18, no. 24, pp.10724–10734, 1998.

[14] A.-S. Augurelle, A. M. Smith, T. Lejeune, and J.-L. Thonnard, “Impor-tance of cutaneous feedback in maintaining a secure grip during manip-ulation of hand-held objects,” J. Neurophysiol., vol. 89, pp. 665–671,2003.

[15] I. Birznieks, P. Jenmalm, A. W. Goodwin, and R. S. Johansson, “En-coding of direction of fingertip forces by human tactile afferents,” J.Neurosci., vol. 21, pp. 8222–8237, 2001.

[16] P. Jenmalm, S. Dahlstedt, and R. S. Johansson, “Visual and tactileinformation about object-curvature control fingertip forces and graspkinematics in human dexterous manipulation,” J. Neurophysiol., vol.84, no. 6, pp. 2984–2997, 2000.

[17] P. Bach-y-Rita, “Tactile sensory substitution studies,” Ann. NY Acad.Sci., vol. 1013, pp. 83–91, 2004.

[18] G. Shannon, “A comparison of alternative means of providing sensoryfeedback on upper limb prostheses,”Med. Biol. Eng. Comput., vol. 14,pp. 289–294, 1976.

[19] P. Nohama, A. Lopes, and A. Cliquet, “Electrotactile stimulator forartificia proprioception,” Artif. Organs, vol. 19, no. 3, pp. 225–230,1995.

[20] G. Wang, X. Zhang, J. Zhang, and W. Gruver, “Gripping force sensoryfeedback for a myoelectrically controlled forearm prosthesis,” in IEEEInt. Conf. Intell. Syst. 21st Century, Vancouver, BC, Canada, 1995, pp.501–504.

[21] S. G. Meek, S. C. Jacobsen, and P. P. Goulding, “Extended physio-logic taction: Design and evaluation of a proportional force feedbacksystem,” J. Rehabil. Res. Develop., vol. 26, no. 3, pp. 53–62, 1989.

[22] D. S. Childress, “Closed-loop control in prosthetic systems—Historicalperspective,” Ann. Biomed. Eng., vol. 8, no. 4–6, pp. 293–303, 1980.

[23] C. A. Phillips, “Sensory feedback-control of upper-extremity andlower-extremity motor prostheses,” Crit. Rev. Biomed. Eng., vol. 16,no. 2, pp. 105–140, 1998.

[24] P. Patterson and J. Katz, “Design and evaluation of a sensory feedback-system that provides grasping pressure in a myoelectric hand,” Bull.Prosthetics Res., vol. 29, no. 1, pp. 1–8, 1992.

Page 8: Haptic Feedback Enhances Grip Force Control of sEMG-Controlled Prosthetic Hands in Targeted Reinnervation Amputees

KIM AND COLGATE: HAPTIC FEEDBACK ENHANCES GRIP FORCE CONTROL OF SEMG-CONTROLLED PROSTHETIC HANDS 805

[25] T. A. Kuiken, L. A. Miller, R. D. Lipschutz, K. A. Stubblefield, and G.A. Dumanian, “Prosthetic command signals followingtargeted hyper-reinnervation nerve transfer surgery,” in Proc. IEEE Eng. Med. Biol.27th Annu. Conf., 2005, pp. 7652–7655.

[26] T. A. Kuiken, G. A. Dumanian, R. D. Lipschutz, L. A. Miller, andK. A. Stubblefield, “Targeted reinnervation for enhanced prostheticarm function in a woman with a proximal amputation: A case study,”Lancet, vol. 369, pp. 371–380, Feb. 2007.

[27] T. A. Kuiken, L. A. Miller, R. D. Lipschutz, B. A. Lock, K. S. eld, P. D.Marasco, P. Zhou, and G. A. Dumanian, “The use of targeted musclereinnervation for improved myoelectric prosthesis control in a bilateralshoulder disarticulation amputee,” Prosthetics Orthotics Int., vol. 28,no. 3, pp. 245–253, Dec. 2004.

[28] T.A.Kuiken, “Considerationof nerve-muscle grafts to improve thecon-trol of artificial arms,” J. Technol. Disabil., vol. 15, pp. 105–111, 2003.

[29] T. A. Kuiken, P. D. Marasco, B. A. Lock, R. N. Harden, and J. P. A.Dewald, “Redirection of cutaneous sensation from the hand to the chestskin of human amputees with targeted reinnervation,” Proc. Nat. Acad.Sci., vol. 104, no. 50, pp. 20061–20066, Dec. 2007.

[30] J. W. Sensinger, A. E. Schultz, and T. A. Kuiken, “Examination offorce discrimination in human upper limb amputees with reinnervatedlimb sensation following peripheral nerve transfer,” IEEE Trans.Neural Syst. Rehabil. Eng., vol. 17, no. 5, pp. 438–444, Oct. 2009.

[31] P. D. Marasco, A. E. Schultz, and T. A. Kuiken, “Sensory capacity ofreinnervated skin after redirection of amputated upper limb nerves tothe chest,” Brain, vol. 132, no. 6, pp. 1441–1448, Jun. 2009.

[32] A. E. Schultz, P. D. Marasco, and T. A. Kuiken, “Vibrotactile detectionthresholds for chest skin of amputees following targeted reinnervationsurgery,” Brain Res., vol. 1251, pp. 121–129, Jan. 2009.

[33] K. Kim, J. E. Colgate, J. J. Santos-Munne, A. Makhlin, and M.A. Peshkin, “On the design of miniature haptic devices for upperexteremity prosthetics,” IEEE Trans. Mechatron., vol. 15, no. 1, pp.27–39, Feb. 2010.

[34] L.Beccai, S.Roccella,L.Ascari, P.Valdastri,A.Sieber,M.C.Carrozza,and P. Dario, “Development and experimental analysis of a soft com-pliant tactile microsensor for anthropomorphic artificial hand,” IEEE/ASMETrans.Mechatron., vol. 13, no. 2, pp. 158–168, Apr. 2008.

[35] L. Zollo, S. Roccella, E. Guglielmelli, M. C. Carrozza, and P. Dario,“Biomechatronic design and control of an anthropomorphic artificialhand for prosthetic and robotic applications,” IEEE/ASME Trans.Mechatron., vol. 12, no. 4, pp. 418–429, Aug. 2007.

[36] P. Gysin, T. R. Kaminski, andA.M.Gordon, “Coordination of fingertipforces in object transport during locomotion,” Exp. Brain Res., vol.149, no. 3, pp. 371–379, 2003.

[37] A. M. Gordon and S. V. Duff, “Fingertip forces during object ma-nipulation in children with hemiplegic cerebral palsy. I: Anticipatoryscaling,” Develop. Med. Child Neurol., vol. 41, no. 3, pp. 166–175,1999.

[38] P. Jenmalm, A. W. Goodwin, and R. S. Johansson, “Control of graspstability when humans lift objects with different surface curvatures,” J.Neurophysiol., vol. 79, no. 4, pp. 1643–1652, 1998.

[39] J. B. Hijjawi, T. A. Kuiken, R. D. Lipschutz, L. A. Miller, K. A. Stub-blerfiled, and G. A. Dumanian, “Improved myoelectric prosthesis con-trol accomplished using multiple nerve transfers,” Brain Res., vol. 118,no. 7, pp. 1573–1578, 2006.

[40] S. Bolanowski, G. Gescheider, and R. Verrillo, “Hairy skin: Psy-chophysical chnnels and their physiological substrates,” Somatosen-sory Motor Res., vol. 11, no. 3, pp. 279–290, 1994.

[41] T. A. Kuiken, G. Li, B. A. Lock, R. D. Lipschutz, L. A. Miller, K. A.Stubblefield, and K. B. Englehart, “Targeted muscle reinnervation forreal-time myoelectric control of multifunction artificial arms,” J. Am.Med. Assoc., vol. 301, no. 6, pp. 619–628, Feb. 2009.

[42] F. V. G. Tenore, A. Ramos, A. Fahmy, S. Acharya, R. Etienne-Cum-mings, and N. V. Thakor, “Decoding of individuated finger movementsusing surface electromyography,” IEEE Trans. Biomed. Eng., vol. 56,no. 5, pp. 1427–1434, May 2009.

[43] J. W. Bisley, A. W. Goodwin, and H. E. Wheat, “Slowly adapting typei afferents from the sides and end of the finger respond to stimuli on thecenter of the fingerpad,” J. Neurophysiol., vol. 84, pp. 57–64, 2000.

[44] P. Jenmalm, I. Birznieks, A. W. Goodwin, and R. S. Johansson, “In-fluence of object shape on responses of human tactile afferents underconditions characteristic of manipulation,” Eur. J. Neurosci., vol. 18,pp. 164–176, 2003.

[45] L. A.Miller, R. D. Lipschutz, K. A. Stubblefield, B. A. Lock, H. Huang,T. W. Williams, R. F. Weir, and T. A. Kuiken, “Control of a six degreeof freedom prostheci arm after targeted muscle reinnervation surgery,”Arch. Phys. Med. Rehabil., vol. 89, pp. 2057–2065, Nov. 2008.

[46] L. A. Miller, K. A. Stubblefield, R. D. Lipschutz, B. A. Lock, and T. A.Kuiken, “Improved myoelectric prosthesis control using targeted rein-nervation surgery: A case series,” IEEE Trans. Neural Syst. Rehabil.Eng., vol. 16, no. 1, pp. 46–50, Feb. 2008.

Keehoon Kim received the B.S., M.S., and Ph.D. de-grees in mechanical engineering from Pohang Uni-versity of Science and Technology (POSTECH), Po-hang, Korea, in 1999, 2001, and 2006, respectively.He was a visiting student at the CaseWestern ReserveUniversity, Cleveland, OH, from 2003 to 2004.He worked as a Postdoctoral Researcher in the De-

partment of Mechanical Engineering, NorthwesternUniversity, Evanston, IL, from 2006 to 2009. He iscurrently a Senior Research Scientist in the Interac-tion and Robotics Research Center, Korea Institute

of Science and Technology (KIST), and an Assistant Professor in University ofScience and Technology (UST). His research interests are robotics in bio-med-ical applications including rehabilitation robotics, surgical robotics, power as-sistant robotics and bionics, haptic interfaces, and teleoperation.

J. Edward Colgate received the Ph.D. degree in me-chanical engineering from the Massachusetts Insti-tute of Technology, Cambridge, in 1988.He subsequently joined Northwestern University,

Evanston, IL, where he is currently the Breed Univer-sity Professor in the Department of Mechanical En-gineering. His principal research interest is human-robot interaction. He has worked extensively in theareas of haptic interface and teleoperation, and he,along with collaborator Michael Peshkin, is the in-ventor of a class of collaborative robots known as

cobots. In addition to his academic pursuits, he is a founder of three companiesincluding Stanley Cobotics, the leading supplier of intelligent ergonomic assistdevices to the industrial marketplace, and Tangible Haptics, a startup focusingon surface haptics. He also directs the Master of Science in Engineering Designand Innovation program, which combines graduate-level engineering courseswith a broad exposure to human-centered design. He has served as an AssociateEditor of the Journal of Dynamic Systems, Measurement and Control.Dr. Colgate has served as an Associate Editor of the IEEE TRANSACTIONS

ON ROBOTICS AND AUTOMATION and he is the founding Editor-in-Chief of theIEEE TRANSACTIONS ON HAPTICS.