““Human Control of Human Control of an Anthropomorphic Robot Hand”an Anthropomorphic Robot Hand”

Aggeliki Tsoli

Odest Chadwicke Jenkins

Brown University, CS Department

Motivation: Lost physical Motivation: Lost physical functionality restorationfunctionality restoration

• Prosthetic handsProsthetic hands human (brain) controlledhuman (brain) controlled anthropomorphicanthropomorphic

www.amputee-coalition.org

Sparse Control ProblemSparse Control Problem

• High-dimensional robot High-dimensional robot handhand up to 30 DOFsup to 30 DOFs

• Low-dimensional human Low-dimensional human inputinput neural decoding: neural decoding:

2-3 DOFs 2-3 DOFs

• 1-to-1 mapping not 1-to-1 mapping not enough!enough!

http://www.faulhaber-group.com

ApproachApproach

Manifold Learning

What is manifold learning?What is manifold learning?• ManifoldManifold

Topological space locally EuclideanTopological space locally Euclidean

• Manifold Learning (i.e. Dimension Reduction)Manifold Learning (i.e. Dimension Reduction) Derive model of d-dimensional manifold embedded in D-Derive model of d-dimensional manifold embedded in D-

dimensional space, d << Ddimensional space, d << D

Methods based on proximity graphs (i.e. neighborhood Methods based on proximity graphs (i.e. neighborhood graphs)graphs)

2D manifold in 3D space2D manifold in 3D space unfolded 2D manifoldunfolded 2D manifold

http://www.seas.upenn.edu/~kilianw/nldrworkshop06

Neighborhood graphs issueNeighborhood graphs issue

• SolutionSolution: Neighborhood Denoising!: Neighborhood Denoising!

Noisy neighborhood graphNoisy neighborhood graph EmbeddingEmbedding

EmbeddingEmbeddingNoise-free neighborhood graphNoise-free neighborhood graph

BP-Isomap: ProcedureBP-Isomap: Procedure

1.1. Neighborhood graph from D-dimensional Neighborhood graph from D-dimensional input datainput data

k-NN or ε-radius ballk-NN or ε-radius ball

2.2. Neighborhood DenoisingNeighborhood Denoising

3.3. Shortest paths for all vertex pairs Shortest paths for all vertex pairs

4.4. d-dimensional embedding using Multi-d-dimensional embedding using Multi-Dimensional Scaling (MDS) Dimensional Scaling (MDS) [Torgerson 1952][Torgerson 1952]

preserve shortest-path distances in embeddingpreserve shortest-path distances in embedding

Step 2: Neighborhood denoisingStep 2: Neighborhood denoising

• Neighbors may not represent correct Neighbors may not represent correct distances along manifolddistances along manifold latent distancelatent distance

Denoising ProcedureDenoising Procedure

A.A. Markov Random Field (MRF) model on Markov Random Field (MRF) model on neighborhood graphneighborhood graph

B.B. Belief PropagationBelief Propagation [Yedidia et al. 2003][Yedidia et al. 2003] (BP) on (BP) on MRFMRF

C.C. Use threshold to identify noisy edgesUse threshold to identify noisy edges

A. Markov Random Field (MRF) A. Markov Random Field (MRF) modelmodel

• For each edge ijFor each edge ij xxijij : latent distance (random variable) : latent distance (random variable) yyijij : observed (Euclidean) distance : observed (Euclidean) distance

• 2 types of functions2 types of functionsa.a. local evidence functionlocal evidence function

b.b. compatibility functioncompatibility function

neighborhood graphneighborhood graph MRF overMRF overneighborhood graphneighborhood graph

a. Local Evidence functiona. Local Evidence function

• Latent distance xLatent distance xijij close to observed close to observed distance ydistance yijij

b. Compatibility functionb. Compatibility function• Correlates the latent distances of adjacent edgesCorrelates the latent distances of adjacent edges

• Check non-common verticesCheck non-common vertices Euclidean distance good indication of latent distanceEuclidean distance good indication of latent distance

Possible noisy edgesPossible noisy edges

B. Belief Propagation on MRFB. Belief Propagation on MRF• Until convergence,Until convergence,

– Select random neighboring edges Select random neighboring edges jm, ijjm, ij– Message from Message from jmjm to to ijij

– Latent distance probability distribution (belief) in Latent distance probability distribution (belief) in ijij Discrete belief over given distancesDiscrete belief over given distances

jm ijm

ij

C. Threshold to determine C. Threshold to determine noisy edgesnoisy edges

• For each edge ijFor each edge ij Pick mode v of latent distance xPick mode v of latent distance x ijij

Noisy edge: v > T Noisy edge: v > T

Result I:Noisy 3D Swiss RollResult I:Noisy 3D Swiss Roll

noisy neighborhood graphnoisy neighborhood graph denoised neighborhood graphdenoised neighborhood graph

2D embeddings2D embeddings

PCAPCA IsomapIsomap BP-IsomapBP-Isomap

Noisy 3D Swiss Roll EvaluationNoisy 3D Swiss Roll Evaluation

pairsvertex

truthgroundncedistEuclideanembeddingE 2)(

MethodMethod 2D Embedding Error2D Embedding Error

PCAPCA 1.2580 x 101.2580 x 101212

IsomapIsomap 8.7271 x 108.7271 x 101111

BP-IsomapBP-Isomap 1.2099 x 101.2099 x 1088

Result II: Hand motion Result II: Hand motion embeddingembedding

• ~500 frames, 25 DOFs~500 frames, 25 DOFs• tapping - powergrasp - precisiongrasptapping - powergrasp - precisiongrasp

BP-Isomap / Isomap PCA

Motion embedding evaluationMotion embedding evaluation

BP-Isomap LimitationsBP-Isomap Limitations

• Doesn’t handle many adjacent bad linksDoesn’t handle many adjacent bad links• Discrete latent variables xDiscrete latent variables xijij

• Tuning of method parametersTuning of method parameters α, β, T, α, β, T, σσ

Future WorkFuture Work

• Extension for adjacent bad linksExtension for adjacent bad links

• Denoise neighborhoods for ST-Isomap Denoise neighborhoods for ST-Isomap [Jenkins et al. 2004][Jenkins et al. 2004] Predict teleoperation failure from tactile and Predict teleoperation failure from tactile and

force sensor embeddings [Peters, Jenkins 05]force sensor embeddings [Peters, Jenkins 05]

Questions ?Questions ?