Mixed Scale Motion Recovery

James DavisJames Davis

Ph.D. Oral PresentationPh.D. Oral Presentation

Advisor – Pat HanrahanAdvisor – Pat Hanrahan

Aug 2001Aug 2001

2

High level goal

• Recover motionRecover motion• Large working volume

• Extreme detail

3

• Acquire real motion as computer modelAcquire real motion as computer model

• Fixed resolution vs. range ratioFixed resolution vs. range ratio

Current technology

4

Applications of motion recovery

• AnimationAnimation

• Athletic analysisAthletic analysis

• BiomechanicsBiomechanics

5

Mixed scale domains

• Detailed motion within a larger volumeDetailed motion within a larger volume

6

Problem domain characterization

• Multiple scales of motionMultiple scales of motion

• At individual scalesAt individual scales• Working volume is local

• Working volume is moving

7

Hierarchical paradigm

• Explicitly expresses motion hierarchyExplicitly expresses motion hierarchy• Motion recovery drives sub-region selection

• Sub-region selection defines next scale

• Multiple designs possible within frameworkMultiple designs possible within framework

Motion recovery

Sub-region selection

Motion recovery

Motion recovery

Sub-region selection

Large scale Medium scale Small scale

8

My design• Multi-camera large scale recoveryMulti-camera large scale recovery

• Covers large volume

• Robust to occlusion

• Pan-tilt multi-camera small scale recoveryPan-tilt multi-camera small scale recovery• Automated camera control

• High resolution imaging

9

Demonstration of my system

• Body moves on room size scaleBody moves on room size scale

• Face deforms on much smaller scaleFace deforms on much smaller scale

• Simultaneous captureSimultaneous capture

10

Desirable system properties

• High resolution/range ratioHigh resolution/range ratio

• Scalable Scalable

• Occlusion robustnessOcclusion robustness

• Runtime automationRuntime automation

11

Related Work

• Traditional motion recoveryTraditional motion recovery• [Vicon] [MotionAnalysis] [Guenter98]

• Simple pan/tilt systemsSimple pan/tilt systems• [Sony EVI-D30] [Fry00]

• 2D guided pan/tilt systems2D guided pan/tilt systems• [Darrell96] [Greiffenhagen00]

• Human controlled camerasHuman controlled cameras• [Kanade00]

Runtime automation

High resolution/range ratio

Scalable

Occlusion robustness

Recovers motion

12

Contributions

• Framework for mixed scale motion recoveryFramework for mixed scale motion recovery• Hierarchical paradigm

• Data driven analysis

• Model based solutions

• Specific system designSpecific system design• High resolution/range ratio

• Scalable

• Robust to occlusion

• Automated

• Application to simultaneous face-body captureApplication to simultaneous face-body capture

13

Talk outline

• IntroductionIntroduction

• Framework Framework • Hierarchical paradigm

• Data driven analysis

• Model based solutions

• System implementationSystem implementation• Large scale recovery

• Sub-region selection

• Small scale recovery

• End-to-end videoEnd-to-end video

• Summary and future workSummary and future work

14

Relation of system to hierarchy

Large scalemotion recovery

Small scalemotion recovery

Sub-region selection

15

System overview

Feature Tracking

3D Pose Recovery

Video streams

2D points

Pan/tilt parameters

Pan/tilt controller

LEDs

3D points

Feature Tracking

3D Pose Recovery

Video streams

2D points

3D points

Large scalemotion recovery

Small scalemotion recovery

Sub-region selection

16

Interface is the challenge

Feature Tracking

3D Pose Recovery

Video streams

2D points

Pan/tilt parameters

Pan/tilt controller

LEDs

3D points

Feature Tracking

3D Pose Recovery

Video streams

2D points

3D points

• Large/small scale similarLarge/small scale similar

• Interface requirements differInterface requirements differ

• Interface critical in end-to-end systemInterface critical in end-to-end system

• Often ignored in individual componentsOften ignored in individual components

Interfaces

17

Data flow

Feature Tracking

3D Pose Recovery

Video streams

2D points

Pan/tilt parameters

Pan/tilt controller

LEDs

3D points

Feature Tracking

3D Pose Recovery

Video streams

2D points

3D points

• System viewed as data flowSystem viewed as data flow

• Clean interface (data) desirable Clean interface (data) desirable

18

System challenges

Feature Tracking

3D Pose Recovery

Video streams

2D points

Pan/tilt parameters

Pan/tilt controller

LEDs

3D points

Feature Tracking

3D Pose Recovery

Video streams

2D points

3D points

Occlusion

Noise

Unknown correspondence

Incorrect camera model

Latency

Occlusion

Unknown camera motion

19

Model improves data

Feature Tracking

3D Pose Recovery

Video streams

2D points

Pan/tilt parameters

Pan/tilt controller

LEDs

3D points

Feature Tracking

3D Pose Recovery

Video streams

2D points

3D points

Occlusion

Noise

Unknown correspondence

Incorrect camera model

Latency

Occlusion

Unknown camera motion

Model

Model

Model

Model

Model

Model

20

Model improves data

Feature Tracking

3D Pose Recovery

Video streams

2D points

Pan/tilt parameters

Pan/tilt controller

LEDs

3D points

Feature Tracking

3D Pose Recovery

Video streams

2D points

3D points

Occlusion

Noise

Unknown correspondence

Incorrect camera model

Latency

Occlusion

Unknown camera motion

Kalman filter

P/T camera model

Prediction

Face model

21

Talk outline

• IntroductionIntroduction

• Framework Framework • Hierarchical paradigm

• Data driven analysis

• Model based solutions

• System implementationSystem implementation• Large scale recovery

• Sub-region selection

• Small scale recovery

• End-to-end videoEnd-to-end video

• Summary and future workSummary and future work

22

Large scale system

Feature Tracking

3D Pose Recovery

Video streams

2D points

Pan/tilt parameters

Pan/tilt controller

LEDs

3D points

Feature Tracking

3D Pose Recovery

Video streams

2D points

3D points

23

Large scale physical arrangement

• 18 18 NTSCNTSC cameras cameras

• 18 digitizing Indys18 digitizing Indys

24

Large scale features

25

Large scale pose recovery

• Consider rays through observationsConsider rays through observations

• Rays cross at 3D feature pointsRays cross at 3D feature points

26

Calibrating wide area cameras

• Jointly calibrated multiple camerasJointly calibrated multiple cameras

• Iteratively estimateIteratively estimate• Camera calibration

• Target path

[Chen, Davis 00]

27

Unknown correspondence

Feature Tracking

3D Pose Recovery

Video streams

2D points

Pan/tilt parameters

Pan/tilt controller

LEDs

3D points

Feature Tracking

3D Pose Recovery

Video streams

2D points

3D points

Unknown temporal correspondenceKalman filter

28

Unknown temporal correspondence

• Multiple 3D features recoveredMultiple 3D features recovered

• Which feature is the head?Which feature is the head?

• Each frame is independently derivedEach frame is independently derived

29

Dynamic motion model

• Not single frame triangulationNot single frame triangulation

• Dynamic motion modelDynamic motion model• Model continuous motion

• Update on each observation

• Estimate position/velocity

• Extended Kalman filter

[Kalman 60] [Broida86] [Welch97]

30

Benefits of motion model

• Feature Feature IDsIDs maintained maintained

• Robust to short occlusionRobust to short occlusion

• Synchronized cameras unnecessary Synchronized cameras unnecessary

31

Sub-region selection

Feature Tracking

3D Pose Recovery

Video streams

2D points

Pan/tilt parameters

Pan/tilt controller

LEDs

3D points

Feature Tracking

3D Pose Recovery

Video streams

2D points

3D points

32

Simplistic camera model

Feature Tracking

3D Pose Recovery

Video streams

2D points

Pan/tilt parameters

Pan/tilt controller

LEDs

3D points

Feature Tracking

3D Pose Recovery

Video streams

2D points

3D points

Incorrect camera model

P/T camera model

33

Simplistic camera model

• Pan/tilt axes not aligned with optical centerPan/tilt axes not aligned with optical center

IxIy

= C Ry Rx

xyz

34

New camera model

• Arbitrary pan/tilt axesArbitrary pan/tilt axes

• Jointly calibrate axes and cameraJointly calibrate axes and camera• Observe known points from several pan/tilt settings

• Fit data with minimum error

= C Tpan Rpan Tpan Ttilt Rtilt Ttilt

IxIy

xyz

-1-1

[Shih 97]

35

Latency

Feature Tracking

3D Pose Recovery

Video streams

2D points

Pan/tilt parameters

Pan/tilt controller

LEDs

3D points

Feature Tracking

3D Pose Recovery

Video streams

2D points

3D points

LatencyPrediction

36

Camera motor latency

• Empirically found 300ms latencyEmpirically found 300ms latency

• High velocity targets leave frameHigh velocity targets leave frame

• Prevents accurate sub-region selectionPrevents accurate sub-region selection

37

Target motion prediction

• Predict future target motionPredict future target motion

• Point camera at predicted target locationPoint camera at predicted target location

• Use previous motion modelUse previous motion model

• High velocity objects successfully trackedHigh velocity objects successfully tracked

P’ = Pi + t • Vi

38

Small scale system

Feature Tracking

3D Pose Recovery

Video streams

2D points

Pan/tilt parameters

Pan/tilt controller

LEDs

3D points

Feature Tracking

3D Pose Recovery

Video streams

2D points

3D points

39

Small scale physical arrangement

• 4 Pan-tilt cameras point at sub-region4 Pan-tilt cameras point at sub-region

• 4 SGI O2s digitize video4 SGI O2s digitize video

40

Small scale features

• Painted face featuresPainted face features

• Image gradient feature trackingImage gradient feature tracking

[Lucas,Kanade 81] [Tomasi, Kanade 91]

41

Small scale pose recovery

42

Problems with face recovery

Feature Tracking

3D Pose Recovery

Video streams

2D points

Pan/tilt parameters

Pan/tilt controller

LEDs

3D points

Feature Tracking

3D Pose Recovery

Video streams

2D points

3D points

Occlusion

Unknown camera motion

Face model

43

Problems with face recovery

• Self occlusionSelf occlusion• Many points not visible

• Camera motion not known preciselyCamera motion not known precisely• Difficult to merge more than two views

Recovered 3D geometryView from one camera

44

Face model

• Face model defines the set of valid facesFace model defines the set of valid faces• Linear combination of basis faces

• Capture basis set under ideal conditions

• Basis transformation

FF = = wwii BBii

= w1 w2 w3+ +

[Turk, Pentland 91] [Blanz,Vetter 99] [Guenter et.al. 98]

45

Model evaluation

Mean error < 1.5 mmMean error < 1.5 mm

Remove

features

Fit to

model

Reconstruct

features

Observe

everything

Evaluate

error

46

Reconstructed face

• Fit partial data to the modelFit partial data to the model• Use model to reconstruct complete geometryUse model to reconstruct complete geometry

Recovered

geometry

from video

Reconstructed

geometry

from model

47

End to end video

48

Talk outline

• IntroductionIntroduction

• Framework Framework • Hierarchical paradigm

• Data driven analysis

• Model based solutions

• System implementationSystem implementation• Large scale recovery

• Sub-region selection

• Small scale recovery

• End-to-end videoEnd-to-end video

• Summary and future workSummary and future work

49

Summary of contributions

• Framework for mixed scale motion recoveryFramework for mixed scale motion recovery• Hierarchical paradigm

• Data driven analysis

• Model based solutions

• Specific system designSpecific system design• High resolution/range ratio

• Scalable

• Robust to occlusion

• Automated

• Application to simultaneous face-body captureApplication to simultaneous face-body capture

50

Future directions

• Application to other domainsApplication to other domains

• More levels of hierarchyMore levels of hierarchy

• Selection of multiple sub-regionsSelection of multiple sub-regions

• Alternate system designsAlternate system designs

51

Acknowledgements

Prof. Pat Hanrahan,Prof. Pat Hanrahan, Prof. Brian Wandell, Prof. Brian Wandell, Prof. Chris Bregler, Prof. Gene Alexander, Prof. Chris Bregler, Prof. Gene Alexander, Prof. Prof. Marc Levoy, Cindy Chen, AdaMarc Levoy, Cindy Chen, Ada Glucksman, Heather Gentner, Homan IgehyGlucksman, Heather Gentner, Homan Igehy, , Venkat Krishnamurthy,Venkat Krishnamurthy, Tamara Munzner, Tamara Munzner, FranFranççois Guimbretiois Guimbretièère,re, Szymon Rusinkiewicz,Szymon Rusinkiewicz, Maneesh Agrawala, Lucas Pereira, Maneesh Agrawala, Lucas Pereira, Kari Pulli, Kari Pulli, Shorty, Sean Anderson, Reid Gershbein, Philipp Shorty, Sean Anderson, Reid Gershbein, Philipp Slusallek, Milton Chen, Mathew Eldridge, Natasha Slusallek, Milton Chen, Mathew Eldridge, Natasha Gelfand, Gelfand, Olaf HallOlaf Hall--Holt, Humper, Brad Johanson, Sergey Brin, Holt, Humper, Brad Johanson, Sergey Brin, Dave Koller, John Owens, Dave Koller, John Owens, Kekoa ProudfootKekoa Proudfoot,, KathyKathy Pullen, Bill Mark, Dan Pullen, Bill Mark, Dan Russel, Larry Page, Li-Yi Wei, Russel, Larry Page, Li-Yi Wei, Gordon Stoll, Julien Gordon Stoll, Julien BaschBasch, , Andrew Beers, Hector Andrew Beers, Hector Garcia-Molina,Garcia-Molina, Brian Freyburger,Brian Freyburger, Mark Horowitz,Mark Horowitz, Erika ChuangErika Chuang, , Chase Garfinkle,Chase Garfinkle, John Gerth, John Gerth,

Xie Feng,Xie Feng, Craig Kolb,Craig Kolb, ToliToli, , Mom, Dad,Mom, Dad, Holly Jones, Chris, Crystal, Lara, Holly Jones, Chris, Crystal, Lara, Grace Gamoso,Grace Gamoso, Matt Hamre,Matt Hamre, Nancy Schaal,Nancy Schaal, Aaron Jones,Aaron Jones, Bandit, Xiaoyuan Bandit, Xiaoyuan TuTu, , Abigail, Shefali,Abigail, Shefali, Liza,Liza, Phil,Phil, Deborah,Deborah, Brianna, Alejandra, Miss Dungan, Gabe, Sedona, Sharon, Gonzalo, and many other children whose names I can no longer rememberBrianna, Alejandra, Miss Dungan, Gabe, Sedona, Sharon, Gonzalo, and many other children whose names I can no longer remember