Balancing Anatomy and Function in a Musculoskeletal Model of Hands

Aaron Blasdel, Yosuke Ikegami, Koh Ayusawa and Yoshihiko Nakamura

Abstract— Musculoskeletal models are effective tools forunderstanding living systems. To ensure proper model function,they must be checked against the literature or specimens.Existing checking methods require cadaver experimentation,highly knowledgeable medical personnel, and/or significanttime. In this paper, we propose a quick and efficient method,called functional consistency checking, for use when theseresources are not available. This method uses the literatureto define a set of mathematical constraints, custom inversedynamics software to interact with the model and its Jacobianin realtime and then evaluates the models consistency with theseconstraints. The method’s usefulness will be demonstrated byconstructing a human hand prototype, performing functionalconsistency checking, and then comparing the original to theoutput using data from a pianist motion capture.

I. INTRODUCTION

Since humans use their hands in most precision tasks,research that is able to describe their behavior is desirable inareas such as sports equipment design [1], product interfacedesign [2], [3], medical procedures with the intention ofrestoring function [4], and realistic visualization [5]. Mus-culoskeletal modeling has also been shown as a useful andeffective method of revealing the internal structure of thehuman body [6] even in real-time [7]. Further examplesinclude; Sueda’s realistic models [5] of the subcutaneoustendons of the hand that allow the skin to be displacedand deformed as they move underneath, Tsang’s HelpingHand model [8] which allows highly accurate simulationof muscle to bone surface interaction in the forearm andhand for use in animation and hospital settings. Muscu-loskeletal simulations have also allowed the realization ofmedical tools for rehabilitation. Such as, Fisk’s work usingcomputational models of the elbow joint [9] and the neural-prosthesis designed by Chadwick for the simulation of armmovements in patients with high level spinal cord injury [10].Business applications also exist, such as the cell phone easeof use research performed by Miyata using the DaibaHand[2] and consumer packaging design research presented byMochimaru [3].

In most sports and performance arts precise positioning ofthe hand is often critical to success. Athletes and performerscould gain a better understanding of their hand movementsfrom a model that reports, in real-time, the activation of theirmuscles relative to the force of their grip on a bat, club or

This work was supported by the Grant-In-Aid for Scientific Research,Category S (20220001), Japan Society for the Promotion of Science.

The authors are with the Graduate School of Information Science andTechnology, Department of Mechano-Informatics, University of Tokyo, 7-3-1 Hongo, Bunkyo-ku, Tokyo 113-8656, Japan{ablasdel, ikegami, ayusawa, nakamura}@ynl.t.u-tokyo.ac.jp

Fig. 1. The Full Prototype model can be seen with the muscles in yellow,ligaments and tendons in blue. The head and chest were added for referencebut all joints in the neck and chest are fused to minimize input requirements.

instrument. For these reasons the hand is the focus of thisresearch.

The main objective of this paper is to detail a methodto quickly and efficiently compare the function of a mus-culoskeletal model to its description in the literature, but itcan be used with any described wire-driven system. Even ifwe can create a highly accurate and detailed model withmany degrees of freedom (DOF), it is at best similar tothe real system due to system approximations. Therefore,we know that models do not always conform to literaturefunction, they are not functionally consistent, so valida-tion is required after modeling. The proposed method canaccelerate this process. With access to cadavers, medicalimagers and/or knowledgeable medical staff there are highlyaccurate methods available [11]. The proposed method isdesigned to require only an anatomy text and a user thatunderstands musculoskeletal models, kinematics, dynamicsand Jacobians. Using the literature and custom realtimemodeling software the user will check if the model complieswith the descriptions in the literature. To test this methodwe will create a prototype and use the proposed method tocheck it. Then we will compare the two models to evaluatethe method’s usefulness.

A. Contributions

1) Detail the creation of a quick and efficient method tocompare the function of a musculoskeletal model tothe same function as described in the literature.

2) Create a musculoskeletal hand model with functionalconsistency.

2012 IEEE International Conference on Robotics and AutomationRiverCentre, Saint Paul, Minnesota, USAMay 14-18, 2012

978-1-4673-1404-6/12/$31.00 ©2012 IEEE 5130

3) Show that models which strive for anatomical precisiondo not necessarily function in a way that is consistentwith constraints derived from the literature.

This paper is organized as follows: First, Sec. II brieflysummarizes the modeling suite and its use to construct themusculoskeletal hand prototype. Sec. III discusses a methodfor checking musculoskeletal models functional consistencyagainst the literature and uses the method on a prototypehand model. Then, Sec. IV describes a pianist motion captureexperiment to visualize the differences between the originalprototype and the checked model. Finally, Sec. V concludesand considers future work.

II. CONSTRUCTION OF PROTOTYPEMUSCULOSKELETAL HAND

The sDIMS suite was used to process the models presentedin this paper. sDIMS is a multi-body and musculoskeletaldynamics computation software system which includes thosebased on [12]–[14]. In this system bones are represented byrigid links, muscles by wires and via-points are placed alongthe wire to change the line of action.

A. Hand Prototype Construction

The bone link data for the prototype was purchased fromthe Viewpoint Corporation. It was created using Computer-ized Tomography (CT) Data from an adult caucasian model.The joints were arranged according to the descriptions foundin the literature [15]. The muscle wires were attached to thecenter point of the muscle’s end point on the correspondingbone links to imitate the function of muscles and ligamentsas indicated in the literature [15], seen in Fig. 1.

All the DOF with a functional angle over 10 degrees wereincluded. However since sDIMS [12]–[14] allows only hingeand spherical joints [14], the joints that require 2 DOF weremodeled using spherical joints. This is the reason that thenumber of DOF exceeds that of the real hand as seen in TableI. This decision was due to function and appearance concernsof constructions consisting of 2 hinge joints and an extra link.Since this model is intended for use with motion capture fullylimiting the joints was not considered necessary. Furthemorethe model contains fewer muscles than a human hand dueto its primary focus on finger function; for this reason somewrist muscles have also been omitted. The lumbrical muscleswere omitted due to their heavy reliance on sliding motionsto create their function and the inability of sDIMS to providea sliding analog. The number of bones in the model was alsoreduced at the wrist joint as shown in Table I. The eight wristbones, as well as the complex joints system between them,were instead approximated by one spherical wrist joint. Therotation of the forearm was also folded into this wrist joint.

B. Via and End point Alteration

To change the action of a muscle in a wire based muscu-loskeletal model we must change their points of connectionto the bones. Moving a via-point as shown in Fig. 2 will in-crease the torque on the joints for a given muscle contractionforce due to the change in each joints moment arm [16]

Fig. 2. Example of via-point structure and movement. When a Via-Pointof a muscle wire is moved it effects the moment arm relative to the jointson either side of the link it is attached to. This can be used to change themuscles contribution to the the rotation of the joints when it is activated.

TABLE ICOMPONENTS OF MUSCULOSKELETAL HAND MODEL

Components Model RealDOF 74 60Muscles 56 76Ligaments 52 >123Bones 23 30

III. METHOD FOR FUNCTIONAL CONSISTENCYCHECKING

The functional consistency of a model is rated by howwell it conforms to a set of consistency constraints derivedby the user. These constraints are most often inequalitystatements, but anything with a binary truth value when giventhe model as input is sufficient. If all the constraints aresatisfied then the model conforms to the function describedin the literature.

A. Creating Consistency Constraints based on literature

To create a set of functional consistency constraints onemust first determine the literature sources to compare against.In this paper we use Grays Anatomy 40th edition [15]. Oncechosen, the text must be analyzed to identify the specificfeatures that are important to mimic in the model.

In the specific case of the human hand prototype theconstraint types fell in to three categories:

1) Joint Alignment,2) Muscle to Joint Bend Direction,3) Relative Torque applied to each joint along a muscle.Many of these functional consistency constraints concern

the relationship of the joint torque to the force of themuscle. One way to directly view this relationship is throughthe Jacobian which is created when calculating the inversedynamics [12]–[14]. The Jacobian transpose JT is a force tojoint torque transformation matrix that relates the vector ofjoint torques τ to the vector of muscle forces f as describedby

τ = JT f (1)

Analyzing these matrix values can reveal valuable infor-mation of the relationship between the force and torquevectors.

The first category consists of inequality constraints thatcompare the position and angle of joints in the fingers.

5131

They ensure that the finger joints bend in a realistic fashionsuch that each finger’s extension and flexing components arealigned properly. The second category consists of constraintswhich ensure the sign of a Jacobian entry JT

[i,j] is positivein some cases and negative in others. The sign indicatesthe direction the joint will bend when the muscle contracts.The third category compares the relative torques applied tojoints along a specific muscle. For example a muscle that,as described in the literature, bends the fingers and later thewrist should, in the neutral position, have a larger Jacobianentry JT

[i,j] at the finger joints than at the wrist.

B. Jacobian Real-time Interaction Tool (JRIT)

Since many of these functional consistency constraints re-quire interaction with the Jacobian values, a method to viewthem while making alterations to the model was needed. TheJRIT interface, seen in Fig. 3, was designed and constructedfor the purpose of speeding up this interaction with theJacobian. Since the Jacobian matrix depends only on thegeometric parameters, such as joint angles, link length, andthe location of via-points that JRIT can visualize the jointtorque when the unit vector of wire tensions is a given [12]–[14]. Its main feature is placing an ellipsoid on the axis ofrotation for each DOF related to the selected muscle, as seenin Fig. 3. These ellipsoids are colored in a manner as todenote the Jacobian value of the related DOF with respectto the currently selected muscle. A joint is colored red for anegative entry, as described in (2), and green for a positiveentry, as described in (3), on a normalized scale of 0.1 to1.0. If the value is close to zero, blue is added as in (4).The intensity of the color is scaled based on its magnitudenormalized to the maximum entry JMAX for the selectedmuscle in (5).

Red =

{0.1 + 0.9(JT

[i,j]/JMAX) if JT[i,j] < 0

0 if JT[i,j] > 0(2)

Grn =

{0 if JT

[i,j] < 0

0.1 + 0.9(JT[i,j]/JMAX) if JT

[i,j] > 0(3)

Blu =

{0.2 if |JT

[i,j]| < 0.0001

0.0 if |JT[i,j]| > 0.0001

(4)

JMAX = |JT[x,j]| such that ∀y|JT

[x,j]| ≥ |JT[y,j]| (5)

C. Functional Checking Speed vs Previous methods

Prior to JRIT and functional consistency checking, thework-flow used to check the full body model began bymotion capturing a subject. Then the full inverse dynamicsand inverse kinematics of this capture were computed toobtain the joint torque and muscle tensions. Next the userneeded to find an inconstancy in the output based on theirknowledge of anatomy. Then the user would edit the modelin a separate program or text file. Finally the fix must beverified by re-running the full inverse dynamics on the newmodel and starting again at the first step [12]–[14].

Fig. 3. This is JRIT displaying a wire representing an extension muscle,in yellow, and the joint ellipsoids at each of the finger DOFs. Two of theDOFs can be seen crossing at the top most knuckle joint. The green colorindicates that when this muscle is activated there will be a torque on thejoints that will cause the joints to extend.

With JRIT the relevant Jacobian values can be viewed inreal time while editing the model because only the musclerelevant portions of the inverse dynamics are calculated andno capture data is required. No fidelity is lost by removingthe unrelated calculations since the Jacobian entries of linksnot attached to the selected muscle are zero. They are thuseither not displayed or the final result to display is zero. Withthis, direct interaction with the data is possible, allowingthe user to adjust joint angles and view the effects in real-time. Using the data gathered from this interaction they canmove the via-points and edit the model to correct problems.Due to these benefits and the fact that for some models andmotion combinations the inverse dynamics can take hours tosolve, the advantage of JRIT is clear. Using the hand modelas a benchmark we uncovered 56 errors in the prototypeand solved them in an average of 45 minutes each. Beforedeveloping the method, errors were uncovered and solved inthe same model in 2 or more hours each.

D. Fixing Conflicts with Functional Consistency Constraints

When creating a model that is intended to function asclose as possible to a real system we must search for theinconsistencies which will inevitably arise by comparinganatomy to functionality. One of the issues encountered withthe model was a joint alignment issue which is picturedin the top third of Fig. 4. It was found in JRIT but sincejoint axis manipulation is not yet supported it was solvedusing 3D Animation’s Maya tool. The bottom four picturesof Fig. 4 depict a joint issue that can arise when a jointbends. The via-points are fixed relative to their respectivelinks and so as the joint rotates the distance between thetwo via-points decreases. If this distance becomes zero andthe rotation continues the sign of the torque applied to thatjoint by the muscle will invert. This is undesirable since suchinversion does not occur in the human hand [15]. This issue

5132

Fig. 4. Left: Prototype; Right: Checked;Top pair: An issue where the finger tip’s joint axis is misaligned.Middle pair: Shows the neutral state of the first finger.Bottom pair: For comparison the same finger as middle but bent.Comparison: In the prototype model the torque on the joint changes, theellipsoid turns green, when bent only slightly while the checked modelallows for significant bending without changing sign.

occurs more often when there are a large number of DOFsand via-points in a small area, as with a hand model. Theseissues were solved by widening the distance between the via-points in question and moving them further from the linksto minimize crossing.

The final issue relates to the relative strength of therelationship between joints for a specific muscle. As seenin Fig. 5, the first branch of the muscle’s relationship tothe wrist and elbow joints, numbered 4-12, was reduced.The relevant finger extension joints, 30-32, have increasedin value. Examples of some of the functional consistencyconstraints on the Jacobian entries JT

[muscle,joint] relevant tothis muscle are seen in (6) and (7).

|JT[Extension Muscle, Finger]| ≥ |J

T[Extension Muscle, Wrist]| (6)

|JT[Extension Muscle, Wrist]| ≥ |J

T[Extension Muscle, Elbow]| (7)

IV. PIANO PLAYING EXPERIMENTSThe goal of this motion capture is to demonstrate the use-

fulness of the presented method by comparing the prototypeto the model checked by the method discussed in Sec. III.

A. Equipment Setup and Layout

For this motion capture a pianist’s finger and handmovements as well as shoulder position were recorded.The following equipment was used: for motion capture:eight Raptor-4 cameras (by Motion Analysis), and 67 smallmarkers (2mm); for selected muscle activation recording:16 Electromyography (EMG) sensor units (by DELSYS);for sound and key press recording, a Roland SoundCanvasSK-88Pro/G Piano, a Roland FA-101 audio interface, and aMac book pro; for video documentation: 2 high speed videocameras (Canon, Nikon); for floor contact: a force plate (byKistler). All equipment was organized as shown in Fig. 6.The markers were attached as described by Chang et. al [17]as seen in Fig. 7 and used as input for the inverse kinematics.The EMG units were placed to provide muscle activity whichwas used as input for the inverse dynamics [7].

Fig. 5. Left most Joints are nearer the shoulder.The green prototype dots show the large relationship between the muscleand the wrist joint (10-12) and elbow joint (4-9). The established checkconstraints state that the absolute value of these entries should be less thanthe absolute value of the finger joint entries (30-32). The checked modelvalues, red diamonds, show that this consistency constraint holds.

Fig. 6. Piano Capture Layout and Floor plan.The cameras were arranged in a rough oval around the subject in an attemptto minimize marker occlusion. The Speakers and Mic are in place for subjectfeedback and are otherwise unnecessary to the capture.high = 2m tall tripod, mid = 1m tall tripod

Fig. 7. The markers were attached in a manner as to minimize fingersimilarity as well as provide enough data points to reliably capture all DOFs[17]. The EMG units were attached on the forearm to the flexor and extensormuscles as well as the muscles of the thumb and fourth finger in the hand.

5133

B. Visualization

The subject played classical music at different speeds. Itis difficult to prepare and measure force sensors on everykey at the same time. Therefore, we assume that the forcecan be estimated from the midi data, or strength of sound,output from the keyboard. The parameters in the sound toforce formulation were identified one-by-one beginning fromcalibrated force sensors. The force applied to each key canbe approximated by

ftip = ckeyvmidi (8)

Where vmidi represents the strength of sound and ckey is aconstant corresponding to each key. To estimate the constantwith error of each key ckey we set up a Flexiforce forcesensor to obtain the force fcalib on the key while recordingthe midi velocity vcalib and averaged n trials, namely

ckey = (1/n)

n∑k=1

(fkcalib/vkcalib) (9)

With the identified constant, the estimated force ftip oneach key is given by

ftip = ckeyvmidi (10)

ftip was used in the visualizations as the contact force atthe fingertips. They provide experimental data to compare thetwo models and demonstrate the method. You can visualizethe data with the 3D rendering (Fig. 8) and the graphs (Fig.9).

C. Motion Capture Results

In Fig. 8 the lack of muscle activations in the prototypemodel, shown on the left, is clearly evident due to theabsence of activated, red, muscles on the underside of thefingers which would apply force to the keys. In contrast thechecked model shows activating muscles which correspondto the force arrows of the key presses. Further, the graphsof selected muscle forces are shown in Fig. 9. The first twographs show the force data for the two muscles primarilyresponsible for flexing the first finger of the right hand. Inthis time interval the keys are pressed twenty times andin the checked model the force curves can be clearly seento coincide with muscle activations. Additionally, the twocomplimentary muscles activity patterns align properly.

Due to the pianist’s use of the side of the thumb topress the key, the muscles which typically act to move thethumb sideways are active when playing. In Fig. 10 thetwo opposing muscles of the right thumb can be viewed.While not as clearly defined as in the first finger case, thealignment of the sixteen key presses in this time intervalis easily seen by comparing the activations in the muscleactivation graph (middle) to the force graph (top). When theabductor muscle is activated the adductor muscle falls closeto zero, as expected for opposing muscles which are movingcontinuously.

Fig. 8. Left Pair: Prototype; Right Pair: Checked modelRed muscles indicate muscle activation, arrows show key contact force. Itcan be seen that the muscles on the underside of the fingers are not activatedfor the prototype model while they are in the checked model. This behavioris due to the alignment issues and via-point placements solved throughthe presented method. All images are from the same instant in time fromdifferent angles.

V. CONCLUSION

A tool (JRIT) and a method to accelerate the process ofchecking musculoskeletal models, and wire driven modelsin general, against the literature were presented. Through apianist’s motion capture it was shown that this method isuseful and can produce functionally consistent models. Theproposed method gave feedback to improve the model inorder to enhance its functional consistency. We have verifiedthe reliability of the checked model’s output by comparingthe interactive components to the literature of the hand jointand muscle relations. It has been shown that this method canbe a useful tool for more quickly comparing the capabilitiesof musculoskeletal models against the literature.

Furthermore, it has been shown that a model designedwith best effort and intention to achieve anatomical precisionmay not produce the best functional result so validationis required. One possible cause is approximations in themodeling system used, in our case sDIMS. Therefore thedesired function must be balanced with the desired anatomi-cal appearance. This can be achieved by using the presentedmethod that allows checking the Jacobian entries, whichrelate the muscle forces to joint torques, in real time.

The following applications are considered as future work:1) Full Body Model Integration: To explore whether it is

possible to integrate the hand model constructed in thiswork with the full body model [12]–[14].

2) Apply to Other Models: The method presented couldbe used to compare the full body model or a foot modelto the corresponding description in the literature withthe aim to enhance their functional consistency.

3) Complete Hand Model Verification: The functionallychecked model could be compared to established mod-els by comparing force outputs on known manipulationtasks.

5134

Fig. 9. Graphs of Muscle force of the Flexor Superficialis and Profundus.Green: Fingertip Force; Red: Prototype; Blue: Functionally CheckedThe fingertip force clearly aligns with the contraction forces of the flexormuscles. These complimentary muscles activate in sync as expected.

Fig. 10. Graphs of Muscle force of the two thumb muscles.Green: Fingertip Force; Red: Prototype; Blue: Functionally CheckedThe fingertip force clearly aligns with the contraction forces of the abductormuscle. These opposing muscles alternate activation as expected.

4) Statistically Evaluate Time Savings: Perform statisticalanalysis on a large group of users to obtain a moreexact time savings estimate.

VI. ACKNOWLEDGMENTSThe authors gratefully acknowledge the considerable con-

tributions of Taku Kashiwagi and Kanade Kubota.

REFERENCES

[1] S. Kawamura, H. Takihara, H. Minamoto, and Hossain Md. Zahid.Influence of sports equipments on human arm. In Eckehard FozzyMoritz and Steve Haake, editors, The Engineering of Sport 6, pages267–272. Springer New York, 2006.

[2] N. Miyata, M. Kouchi, T. Kmhara, and M. Mochimaru. Modeling ofhuman hand link structure from optical motion capture data. 2004IEEE/RSJ International Conference on Intelligent Robots and Systems(IROS) (IEEE Cat. No.04CH37566), pages 2129–2135, 2004.

[3] Miyata N Yoshida Y Aoki K Kawachi K. et al. Mochimaru M,Kouchi M. Dhaiba: Functional human models to represent variationof shape, motion and subjective assessment. 2006 Digital HumanModeling for Design and Engineering Conference 2006-01-2345.,2006.

[4] P. Taylor, J. Esnouf, and J. Hobby. The functional impact of thefreehand system on tetraplegic hand function. clinical results. SpinalCord, 40(11):560–566, 2002.

[5] Shinjiro Sueda, Andrew Kaufman, and Dinesh K. Pai. Musculotendonsimulation for hand animation. ACM Transactions on Graphics,27(3):1, August 2008.

[6] S.L. Delp and J.P. Loan. A computational framework for simulatingand analyzing human and animal movement. Computing in Science& Engineering, 2(5):46–55, 2000.

[7] Akihiko Murai, Kosuke Kurosaki, Katsu Yamane, and YoshihikoNakamura. Musculoskeletal-see-through mirror: Computational mod-eling and algorithm for whole-body muscle activity visualization inreal time. Progress in biophysics and molecular biology, 103(2-3):310–317, September 2010.

[8] Winnie Tsang and Karan Singh. Helping hand: an anatomicallyaccurate inverse dynamics solution for unconstrained hand motion.Proceedings of the 2005 ACM, pages 29–31, July 2005.

[9] Justin P Fisk and Jennifer S Wayne. Development and validationof a computational musculoskeletal model of the elbow and forearm.Annals of biomedical engineering, 37(4):803–12, April 2009.

[10] Edward K Chadwick, Dimitra Blana, Antonie J Ton van den Bogert,and Robert F Kirsch. A real-time, 3-D musculoskeletal model fordynamic simulation of arm movements. IEEE transactions on bio-medical engineering, 56(4):941–8, April 2009.

[11] S.S. Blemker, D.S. Asakawa, G.E. Gold, and S.L. Delp. Image-based musculoskeletal modeling: applications, advances, and futureopportunities. Journal of Magnetic Resonance Imaging, 25:441–51,2007.

[12] Y. Nakamura and K. Yamane. Dynamics computation of structure-varying kinematic chains and its application to human figures.Robotics and Automation, IEEE Transactions on, 16(2):124 –134, apr2000.

[13] Katsu Yamane and Yoshihiko Nakamura. Natural motion animationthrough constraining and deconstraining at will. IEEE Transactionson Visualization and Computer Graphics, 9:352–360, 2003.

[14] Yoshihiko Nakamura, Katsu Yamane, Yusuke Fujita, and IchiroSuzuki. Somatosensory computation for man-machine interface frommotion-capture data and musculoskeletal human model. IEEE Trans-actions on Robotics, 21(1):58–66, February 2005.

[15] S. Standring and H. Gray. Gray’s anatomy: the anatomical basis ofclinical practice. Gray’s Anatomy: The Anatomical Basis of ClinicalPractice. Churchill Livingstone/Elsevier, 2008.

[16] Saikat Pal, Joseph E Langenderfer, Joshua Q Stowe, Peter J Laz,Anthony J Petrella, and Paul J Rullkoetter. Probabilistic modeling ofknee muscle moment arms: effects of methods, origin-insertion, andkinematic variability. Annals of Biomedical Engineering, 35(9):1632–1642, 2007.

[17] Lillian Y. Chang, Nancy S. Pollard, Tom M. Mitchell, and Eric P.Xing. Feature selection for grasp recognition from optical markers.2007 IEEE/RSJ International Conference on Intelligent Robots andSystems, pages 2944–2950, October 2007.

5135