Interaction Capture and Synthesis -...

3/31/2007

Interaction Captureand

Synthesis

Dinesh K. Pai

University of British Columbia

Rutgers University

[with Paul G. Kry]

Haptics ≡ The sense of touch≡ Force sensing and display?

Haptics ≡ ContactContact is multisensorywith forcessoundsdeformationmotion discontinuities…

Primary VisualCortex

Primary AuditoryCortex

SomatosensoryCortex

MotorCortex

3/31/2007

The AHI Audio-HapticInterfaceThe AHI Audio-HapticInterface

DiFilippo and Pai UIST 00See also [Doel and Pai 96], [Doel, Kry and Pai, SIGGRAPH 01]

3/31/2007

Precomputed Acoustic Transfer[James,Barbic,Pai SIGGRAPH 2006]Precomputed Acoustic Transfer[James,Barbic,Pai SIGGRAPH 2006]

Motivation: Whole hand interaction with the TangoMotivation: Whole hand interaction with the Tango

[Pai, et al. World Haptics 05]

• Whole hand passive haptic interface

• 32 x 8 capacitive sensor, 3 axis accelerometer

3/31/2007

Motivation: Motion capture animation techniques difficult to use with contactMotivation: Motion capture animation techniques difficult to use with contact

Motivation: Whole hand interaction with the TangoMotivation: Whole hand interaction with the Tango

3/31/2007

MotivationMotivation

• Need to capture how humans move– Impedance (stiffness) of muscles…

– Feed-forward control

• Brain actively controls the passive behavior of the musculoskeletal system – Depends on task, geometry, intent, style

– Important for stability

[Feldman 66; Hogan 84; Bizzi et al. 92; etc.]

originalobject

analysis

motion capture

force capture

simulationInteractionTrajectory

newobject

intendedmotion

passivebehavior

Interaction Capture and SynthesisInteraction Capture and Synthesis

newobject

3/31/2007

motion capture cameras 500 Hz

Overview ExampleOverview Example

Overview ExampleOverview Example

motion capture cameras 500 Hz

force-torque sensor 500 Hz

3/31/2007

Overview ExampleOverview Example

Overview ExampleOverview Example

• Estimate the Interaction Trajectory– Compliance (inverse of stiffness)

– Reference trajectory (motion without contact)

analysis

motion capture

force capture

intendedmotion

passivebehavior

referencetrajectory

compliance

3/31/2007

Overview ExampleOverview Example

• Environment– Fully dynamic simulation

• Character – Contact from quasi-static

simulation

– Dynamics from referencetrajectory

analysis

motion capture

force capture

simulation

referencetrajectory

compliance

referencetrajectory

compliance

OutlineOutline

• Interaction trajectory estimation

analysis

motion capture

force capture

simulation

reference trajectory

compliance

3/31/2007

Compliant hand modelCompliant hand model

∆u1

Effective end-point compliance

Compliance is Task DependentCompliance is Task Dependent

Exploring ScratchingWrist 0.35 0.15

MP 0.99 0.47

PIP 0.67 < 0.84

DIP 2.01 < 0.84

• Controlled by the brain in addition to motion– Scratching has half the compliance of exploring

DIP PIP MP Wrist(rad/Nm)

3/31/2007

Compliance EstimationCompliance Estimation

• Previous workArms [Xu, Hollerbach, Hunter 91; Gomi, Kawato 96; etc.]

Fingers [Hajian 97; Milner, Franklin 98; Hasser, Cutkosky 02]

• Stiffness, some measure damping and inertia too

• End points, joints, static poses, during movement

• Approach in common is perturbation– Complicates capture, changes the motion

Compliance Estimation without extra perturbationCompliance Estimation without extra perturbation

• Contact provides a natural perturbation– Exploits slow speed of spinal reflexes

3/31/2007

-0.1 -0.05 0 0.05 0.1

-0.36

-0.34

-0.32

-0.3 joint angle

-0.1 -0.05 0 0.05 0.10

0.02

0.04

0.06 joint torque

seconds

post-contactpre-contact

reference trajectory

post-contact

capture

capture

this joint

Capture Data at time of contactCapture Data at time of contact

seconds

Assumptions & LimitationsAssumptions & Limitations

• Independent compliance estimations

• Natural perturbation must be observable

• Complex ‘pre-programmed’ motion

3/31/2007

ValidationValidation

(rad/Nm) Exploring ScratchingWrist 0.35 0.15

MP 0.99 0.47

PIP 0.67 < 0.84

DIP 2.01 < 0.84

• Estimates show task dependence– Scratching has half the compliance of exploring

DIP PIP MP Wrist

ValidationValidation

• Perturbation via spring loaded platform

3/31/2007

ValidationValidation

seconds

fingertipelevation

normalforce

-0.05 0 0.05 0.1 0.15

0

2

4

-0.05 0 0.05 0.1 0.15

0246

100 ms delay

Estimates reasonable given values in previous studies[Hajian 97; Milner et al. 98; Kuchenbecker et al. 03]

285.9681.86kG (N/m)

257.0489.16k (N/m)

3.862.04b (Ns/m)

0.0180.017m (kg)

1.3450.677f0 (N)

ScratchingExploringParameter

ValidationValidation

kGOUR METHOD

3/31/2007

OutlineOutline

• Interaction Synthesis

analysis

motion capture

force capture

simulation

reference trajectory

compliance

Interaction SynthesisInteraction Synthesis

• Quasi-static simulation– Alternative is fully dynamic simulation

– Capture already contains character’s dynamics

– Focus on synthesizing contact perturbations

– Compute the new forces with Coulomb friction

3/31/2007

• Linear Complimentarity Problem (LCP)– New algorithm for compliant contact and friction

Interaction SynthesisInteraction Synthesis

f k

∆u

∆f

Discretized friction cone Contact Point Motion• Reference trajectory• End point compliance• Dynamic environment

ResultsResults

3/31/2007

ResultsResults

ResultsResults

3/31/2007

Grip AdjustmentGrip Adjustment

• Object unexpectedly heavier than expected– Tighten grip by reducing compliances after a small delay

Grip AdjustmentGrip Adjustment

• Expected changes– Similar motor program, adjusted due to object appearance

– Gradually reduce compliance when slip is imminent

3/31/2007

Example Using Tango CaptureExample Using Tango Capture

analysis

motion capture

force capture

simulation

referencetrajectory

compliance

previously estimated

compliance

Tango captures at 100 Hz

ConclusionsConclusions

• Interaction capture– Extends motion capture to handle contact

– Compliance estimation from natural movement

• Interaction synthesis by simulation– New algorithm for compliant contact with friction

3/31/2007

• Download data at http://www.interactioncapture.org

Thanks to the support of

• NSF grants IIS-0308157, EIA-0215887, ACI-0205671, EIA-0321057

• NIH CRCNS 1R01NS050942

• Canada Research Chair