Kinematic Model of the Hand using Computer VisionEnglish).pdf · Kinematic hand model solver Software to track ﬁngernail markers Publication E. Simo-Serra, F. Moreno-Noguer and

Kinematic Model of the Hand

using Computer Vision

Edgar Simo Serra

Barcelona, Spain

Barcelona - April, 2011

Summary

Obtain hand movements through computervision

Reconstruct hand kinematics from data

Index

1 Overview

2 Kinematic Hand Model

3 Hand Detection

4 Kinematic Synthesis

5 Non-linear solver

6 Results

Overview

Motivation

EMG Signals ⇐⇒ Hand Movement

Mapping of Electromyographic (EMG) signals

Understanding hand movements

Prosthesis design

Exoskeleton design

Objectives

1 Design of a theoretical kinematic model of thehand

2 Detection of real hand characteristics withcomputer vision

3 Solver to adjust the theoretical model to theexperimental data

Kinematic Hand Model

State of the Art

Robotic Grippers

Prosthetic Hands

Match function not form

Simplified kinematic models

State of the Art

Robotic Grippers

Prosthetic Hands



State of the Art

Robotic Grippers

Prosthetic Hands



Hand Anatomy

5 Fingers

14 Joints

27 Bones

Complex articulatedsystem

Hand Topology

3 common joints (wrist)

Two fingers with 4 joints (index, middle)

Three fingers with 5 joints (third, fourth,thumb)

Thumb has different joint structure

Hand Topology

(a) Full hand. (b) Index finger. (c) Middle finger.

(d) Third finger. (e) Fourth finger. (f) Thumb.

Hand Model

26 degrees of freedomAll revolute jointsStrong similarity with bone structure

Detection of Hand Characteristics

State of the Art

Contact approachLow portabilityHigh costNeed for calibrationLow resolution

State of the Art

Contact approachLow portabilityHigh costNeed for calibrationLow resolution

Non-contact ApproachFocus on realtimeLow precisionApplications in augmentedreality

Approach

Tracking of nailsNon-deformableFixed to hand structureEasily identifiableNot easy to identify rotation

Markers for robust designAllows obtaining of pose (position and rotation)

Markers

Simple for robustness

Convey poseinformation

Markers

Simple for robustness

Convey poseinformation

Detecting Markers

1 Detect candidate points with Corner and edgedetector (Harris 1998)

2 Match candidate points with heuristics3 Use DAISY descriptors to do tracking (Tola

2010)

Detecting Markers

1 Detect candidate points with Corner and edgedetector (Harris 1998)

2 Match candidate points with heuristics3 Use DAISY descriptors to do tracking (Tola

2010)

PnP: Pose Estimation Problem

u = [A][R |t]p

Very important problem in computer vision

Identify camera position given n 2D-3Dcorrespondences u ↔ p

Solving the Pose Estimation

Specific case 4 coplanar points: P4P

Single unique solution (Yuan 1989)Usage of combination of algorithms

Orthogonal Iteration (Hager et al. 2000)Robust Pose Estimation (Schweighofer et al. 2006)

Error Analysis

0 50 100 150 200

0

50

100

Pose (0 degrees)

0 50 100 150

0

50

100

Pose (10 degrees)

0 50 100 150

0

50

100

Pose (20 degrees)

0 50 100 150

0

50

100

Pose (30 degrees)

−50 0 50 100 150

0

50

100

Pose (40 degrees)

−50 0 50 100 150

0

50

100

Pose (50 degrees)

−50 0 50 100 150

0

50

100

Pose (60 degrees)

−50 0 50 100 150

0

50

100

Pose (70 degrees)

−50 0 50 100 150

0

50

100

Pose (80 degrees)

Error Analysis

0 5 10 150

50

100

Noise Standard Deviation (px)

Err

or

(pe

rce

nt)

T. Error (0 degrees)

Interval (99%)

Mean

0 5 10 150

50

100


Err

or

(pe

rce

nt)


0 5 10 150

50

100


Err

or

(pe

rce

nt)


0 5 10 150

50

100


Err

or

(pe

rce

nt)


0 5 10 150

50

100


Err

or

(pe

rce

nt)


0 5 10 150

50

100


Err

or

(pe

rce

nt)


0 5 10 150

50

100


Err

or

(pe

rce

nt)


0 5 10 150

50

100


Err

or

(pe

rce

nt)


0 5 10 150

50

100

Noise Standard Deviation (px)E

rro

r (p

erc

en

t)


Error Analysis

0 5 10 150

20

40

60


Ro

tatio

n E

rro

r (p

erc

en

t)

R. Error (0 degrees)

Interval (99%)

Mean

0 5 10 150

20

40

60


Ro

tatio

n E

rro

r (p

erc

en

t)


0 5 10 150

20

40

60


Ro

tatio

n E

rro

r (p

erc

en

t)


0 5 10 150

20

40

60


Ro

tatio

n E

rro

r (p

erc

en

t)


0 5 10 150

20

40

60


Ro

tatio

n E

rro

r (p

erc

en

t)


0 5 10 150

20

40

60


Ro

tatio

n E

rro

r (p

erc

en

t)


0 5 10 150

20

40

60


Ro

tatio

n E

rro

r (p

erc

en

t)


0 5 10 150

20

40

60


Ro

tatio

n E

rro

r (p

erc

en

t)


0 5 10 150

20

40

60

Noise Standard Deviation (px)R

ota

tio

n E

rro

r (p

erc

en

t)


Detection of Hand

Reliable and robust

Low error in pose estimation with little noise

Experimental dataset for adjusting theoreticalhand model

Kinematic Synthesis

Kinematic Synthesis

Generalized inverse kinematics problem

Changing structural parameters to solveproblem

Motion-to-form problem

Very complex and generally unsolved problem

State of the Art

Simplifying the problemGroebner basesElimination theoryLinear product decomposition

State of the Art


Revolute joint structures solved

2R: 2 solutions (A. Perez andJ.M. McCarthy 2003)

State of the Art



2R: 2 solutions (A. Perez andJ.M. McCarthy 2003)3R: 13 solutions in an interval(Lee et al. 2004)

State of the Art



2R: 2 solutions (A. Perez andJ.M. McCarthy 2003)3R: 13 solutions in an interval(Lee et al. 2004)4R: Unsolved

State of the Art



2R: 2 solutions (A. Perez andJ.M. McCarthy 2003)3R: 13 solutions in an interval(Lee et al. 2004)4R: Unsolved...26R: Unsolved (our problem)

Kinematic Chain

F

[G]

S1

S2

S3

a12

d12

a23

[H]

Θ12

α12

α23

M

Articulated system formed by jointsDifferent topologies

SerialParallelLoopTree (hand model)

F

[G]

Common Joints

Branch 2

Branch 1

S1

S2

S3 S4

S41

2

Plucker Coordinates

Line representation: S = s︸︷︷︸

orientation

+ǫ s0︸︷︷︸

moment

Only 2 representations per line (s and −s)

Dual Quaternions

Homogeneous Matrix [H]︸︷︷︸

16 components

7−→ Dual Quaternion Q︸︷︷︸

8 components

Element of Clifford algebraUnit dual quaternions

Represents 3D displacements, points and linesTwo constraints: qq∗ = 1 and q · q0 = 0

Exponential Map

Based on Lie Algebra

Allows joints to be written as1-parameter screws

Joint parameters: θ = θ + ǫdPrismatic: θ = 0Revolute: d = 0

d

Θ

S

S(θ) = eθ

2S = cos

θ

2+ sin

θ

2S

Forward Kinematics

Determines end effector position based onjoints

Direct calculation for serial kinematictopologies

T (θ) = S1(θ1)S2(θ2) · · · Sn(θn)G

= eθ12S1e

θ22S2 · · · e

θn

2SnG

= (cosθ1

2+ sin

θ1

2S1) · · · (cos

θn

2+ sin

θn

2Sn)G

Forward Kinematics

∆θ = θ − θref + ǫ(d − dref )

Q(∆θ) = S1(∆θ1)S2(∆θ2) · · · Sn(∆θn)

= e∆θ1

2S1e

∆θ22

S2 · · · e∆θn

2Sn

= (cos∆θ1

2+ sin

∆θ1

2S1)

· · · (cos∆θn

2+ sin

∆θn

2Sn)

Design Equations

Input: task defined by m − 1 relativetransformations P1j = TjT

−11 , j = 2, . . . , m

Variables: joint axes and joint anglesEquations:

S1j(∆θ1j)S2j(∆θ2j)S3j(∆θ3j)

Sk4j(∆θk

4j) · · · Sk7j(∆θk

7j) − Pk1j = 0, k ∈ 1, 2,

S1j(∆θ1j)S2j(∆θ2j)S3j(∆θ3j)

Sk4j(∆θk

4j) · · · Sk8j(∆θk

8j) − Pk1j = 0, k ∈ 3, 4, 5,

j = 2, ..., m

Non-linear Solver

System Dimension

Relations between variables and equationsdecrease dimension

Equations use 3 algebraic sets:

Set Comp Ind Notesθ 1 1 Periodic with period 2π

S = s + ǫs0 6 4 s · s0 = 0 , ‖s‖ = 1

Q = q + ǫq0 8 6 q · q0 = 0 , qq∗ = 1

System Dimension

System unknowns:

n0x = r( 4

︸︷︷︸

Structural

+(m − 1)︸︷︷︸

Joint

)

nx = r(6 + (m − 1))

System functions:

n0f = b6(m − 1)

nf = b8(m − 1)

System Dimension

System must be solvable: n0f ≥ n0

x

m =4r

6b − r+ 1 > 0

6b − r > 0

Multiple solution possibilities

System Reduction

System can be reduced to subsystems that canbe solved

Allows reducing the dimension of the system

b r n0x = n0

f nx nf m Notes5 26 780 832 1092 27 Full hand model2 11 528 550 726 45 (4R, 4R) fingers1 5 120 130 170 21 5R, no common

Solution Bound

High dimensionality of the system

Important non-linearity of the equationsBezout’s theorem: D = d1d2 · · · dn

Serial chain: d = (3r)6 (A. Perez 2003)

di = 216(m−1) , i ∈ 1, 2,

di = 246(m−1) , i ∈ 3, 4, 5,

D =5∏

i=1

di ≈ 101058

Numerical Solver

Global optimizer

Allow overdetermined systems

Numerical Solver

Global optimizer

Allow overdetermined systemsLevenberg-Marquadt Algorithm

Local optimizerLeast squaresVery localVery difficult to parallelize

Numerical Solver

Global optimizer

Allow overdetermined systemsLevenberg-Marquadt Algorithm

Local optimizerLeast squaresVery localVery difficult to parallelize

Genetic Algorithm“Global” optimizer(meta-heuristics)Easy to implementVery hard to tuneExtremely parallel

Implementation

Hybrid solverReduces search space by minimizing withLevenberg-Marquadt algorithmGlobal optimizer by using genetic algorithm

Implementation


Implementation


Results

Solver

Parallel algorithm

Bottleneck in QR decomposition fromLevenberg-Marquadt algorithm

Average run:

Generations Runtime (hours)38 60

Solver

Mutation

Solution

Solutions

Non-anthropomorphic structure

Moves finger positions exactly the same as ahandNeeds support for constraints to distinguish

More complicated solver

Solutions

Non-anthropomorphic structure

Moves finger positions exactly the same as ahandNeeds support for constraints to distinguish

More complicated solver

Solutions



Results

Softwarelibdq: Library for manipulating dual quaternionsKinematic hand model solverSoftware to track fingernail markers

PublicationE. Simo-Serra, F. Moreno-Noguer and A.Perez-Gracia, “Design of non-anthropomorphicrobotic hands for anthropomorphic tasks”,ASME-IDETC, accepted.

Conclusions

Major problems in engineeringPnP or pose estimation problemMotion-to-form or dimensional kinematic synthesisproblem

Results reflect maturity in both problems

Conclusions


Results reflect maturity in both problemsHand detection

Low computation costReliable with slow movements

Conclusions


Results reflect maturity in both problemsHand detection

Low computation costReliable with slow movements

Kinematic synthesisVery high complexityLarge number of solutionsNon-anthropomorphic solutions found

Conclusions

Objectives1 Defined theoretical hand model2 Detected real hand characteristics with computer

vision3 Adjusted theoretical model to experimental data

Conclusions

Objectives1 Defined theoretical hand model2 Detected real hand characteristics with computer

vision3 Adjusted theoretical model to experimental data

Anthropomorphic solution not foundFirst project to attempt dimensional kinematicsynthesis of tree-like topologiesKinematic synthesis is generally unsolved (3R isnot algebraically solved)Upper bound of 101058 solutions

Future Work

Flexible generic tree-like topology dimensionalkinematic synthesis solver

Exchangeable backendsScripting supportSupport for constraints

Future Work

Flexible generic tree-like topology dimensionalkinematic synthesis solver

Exchangeable backendsScripting supportSupport for constraints

Study of design equationsHow many solutions are there?What causes loss of dimensionality?How are solutions (and local minima) distributed?Can the a reduced model of the search space beused for initial estimation?Are matrix representation better at convergence?

Thanks


Questions?


Documents

Kinematic Model of the Hand using Computer VisionEnglish).pdf · Kinematic hand model solver Software to track ﬁngernail markers Publication E. Simo-Serra, F. Moreno-Noguer and