Multiresolution Motion Analysis with Applications Jehee Lee Sung Yong Shin Dept of EE&CS, KAIST...

Multiresolution Motion Analysiswith Applications

Jehee Lee Sung Yong ShinJehee Lee Sung Yong Shin

Dept of EE&CS, KAISTDept of EE&CS, KAIST

Jehee Lee Sung Yong ShinJehee Lee Sung Yong Shin

Dept of EE&CS, KAISTDept of EE&CS, KAIST

Contents

1.1. IntroductionIntroduction

2.2. Multiresolution AnalysisMultiresolution Analysis

3.3. Hierarchical Displacement MappingHierarchical Displacement Mapping

4.4. Spatial Filtering for Motion DataSpatial Filtering for Motion Data

5.5. ApplicationsApplications

1.1. IntroductionIntroduction

2.2. Multiresolution AnalysisMultiresolution Analysis

3.3. Hierarchical Displacement MappingHierarchical Displacement Mapping

4.4. Spatial Filtering for Motion DataSpatial Filtering for Motion Data

5.5. ApplicationsApplications

Character Animation

Realistic motion dataRealistic motion data

• Motion capture technologyMotion capture technology

• Commercial librariesCommercial libraries

Producing animation from available motion clipsProducing animation from available motion clips

• requires specialized toolsrequires specialized tools

– interactive editing, smoothing, enhancement, blending, interactive editing, smoothing, enhancement, blending, stitching, and so onstitching, and so on

• stimulates the need for a unified approachstimulates the need for a unified approach

Realistic motion dataRealistic motion data

• Motion capture technologyMotion capture technology

• Commercial librariesCommercial libraries

Producing animation from available motion clipsProducing animation from available motion clips

• requires specialized toolsrequires specialized tools

– interactive editing, smoothing, enhancement, blending, interactive editing, smoothing, enhancement, blending, stitching, and so onstitching, and so on

• stimulates the need for a unified approachstimulates the need for a unified approach

Multiresolution Analysis

Representing a signal at multiple resolutionsRepresenting a signal at multiple resolutions

• gives hierarchy of successively smoother signalsgives hierarchy of successively smoother signals

• facilitates a variety of signal processing tasksfacilitates a variety of signal processing tasks

Representing a signal at multiple resolutionsRepresenting a signal at multiple resolutions

• gives hierarchy of successively smoother signalsgives hierarchy of successively smoother signals

• facilitates a variety of signal processing tasksfacilitates a variety of signal processing tasks

(3)m(3)m (2)m(2)m (1)m(1)m (0)m(0)m

Previous Work

Image and signal processingImage and signal processing

• Gauss-Laplacian pyramidGauss-Laplacian pyramid [Burt and Adelson [Burt and Adelson

83]83]

• Texture analysis and synthesis, image editing, curve and Texture analysis and synthesis, image editing, curve and surface manipulation, data compression, and so onsurface manipulation, data compression, and so on

Motion synthesis and editingMotion synthesis and editing

• Hierarchical spacetime controlHierarchical spacetime control [Liu, Gortler and Cohen [Liu, Gortler and Cohen

94]94]

• Motion signal processingMotion signal processing [Bruderlin and Williams [Bruderlin and Williams

95]95]

Image and signal processingImage and signal processing

• Gauss-Laplacian pyramidGauss-Laplacian pyramid [Burt and Adelson [Burt and Adelson

83]83]

• Texture analysis and synthesis, image editing, curve and Texture analysis and synthesis, image editing, curve and surface manipulation, data compression, and so onsurface manipulation, data compression, and so on

Motion synthesis and editingMotion synthesis and editing

• Hierarchical spacetime controlHierarchical spacetime control [Liu, Gortler and Cohen [Liu, Gortler and Cohen

94]94]

• Motion signal processingMotion signal processing [Bruderlin and Williams [Bruderlin and Williams

95]95]

Issues in Motion Analysis

Difficulties in handling motion dataDifficulties in handling motion data• Inherent non-linearity of orientation spaceInherent non-linearity of orientation space

Coordinate-invarianceCoordinate-invariance• Independent of the choice of coordinate framesIndependent of the choice of coordinate frames

Difficulties in handling motion dataDifficulties in handling motion data• Inherent non-linearity of orientation spaceInherent non-linearity of orientation space

Coordinate-invarianceCoordinate-invariance• Independent of the choice of coordinate framesIndependent of the choice of coordinate frames

Contents

1.1. IntroductionIntroduction

2.2. Multiresolution AnalysisMultiresolution Analysis

3.3. Hierarchical Displacement MappingHierarchical Displacement Mapping

4.4. Spatial Filtering for Motion DataSpatial Filtering for Motion Data

5.5. ApplicationsApplications

1.1. IntroductionIntroduction

2.2. Multiresolution AnalysisMultiresolution Analysis

3.3. Hierarchical Displacement MappingHierarchical Displacement Mapping

4.4. Spatial Filtering for Motion DataSpatial Filtering for Motion Data

5.5. ApplicationsApplications

Motion Representation

Configuration of articulated figuresConfiguration of articulated figures

• Bundle of motion signalsBundle of motion signals

• Each signal represents time-varying positions and Each signal represents time-varying positions and orientationsorientations

• Rigid transformationRigid transformation

Configuration of articulated figuresConfiguration of articulated figures

• Bundle of motion signalsBundle of motion signals

• Each signal represents time-varying positions and Each signal represents time-varying positions and orientationsorientations

• Rigid transformationRigid transformation

33))()(()( SRqpm ttt , 33))()(()( SRqpm ttt ,

pququqp 1)( )(T , pququqp 1)( )(T ,

33:T RR 33:T RR

Decomposition

Expansion : Expansion : up-sampling followed by smoothingup-sampling followed by smoothing

Reduction : Reduction : smoothing followed by down-samplingsmoothing followed by down-sampling

Expansion : Expansion : up-sampling followed by smoothingup-sampling followed by smoothing

Reduction : Reduction : smoothing followed by down-samplingsmoothing followed by down-sampling

)(nm )(nm

)1( nm )1( nm

)1( nd )1( nd

ReductionReduction ExpansionExpansion

Decomposition and Reconstruction

DecompositionDecomposition

ReconstructionReconstruction

DecompositionDecomposition

ReconstructionReconstruction

)(nm )(nm )1( nm )1( nm )2( nm )2( nm )0(m )0(m

)1( nd )1( nd )2( nd )2( nd )0(d )0(d

)0(m )0(m

)1( nd )1( nd )2( nd )2( nd )0(d )0(d

)(nm )(nm )1( nm )1( nm )2( nm )2( nm

Our Approach

Multiresolution Motion AnalysisMultiresolution Motion Analysis

• Hierarchical displacement mappingHierarchical displacement mapping

– How to representHow to represent

– Displacement mapping Displacement mapping [Bruderlin and Williams 95][Bruderlin and Williams 95]

– Motion warping Motion warping [Popovic and Witkin 95][Popovic and Witkin 95]

• Spatial filtering for motion dataSpatial filtering for motion data

– How to constructHow to construct

– Implement reduction and expansionImplement reduction and expansion

Multiresolution Motion AnalysisMultiresolution Motion Analysis

• Hierarchical displacement mappingHierarchical displacement mapping

– How to representHow to represent

– Displacement mapping Displacement mapping [Bruderlin and Williams 95][Bruderlin and Williams 95]

– Motion warping Motion warping [Popovic and Witkin 95][Popovic and Witkin 95]

• Spatial filtering for motion dataSpatial filtering for motion data

– How to constructHow to construct

– Implement reduction and expansionImplement reduction and expansion

Contents

1.1. IntroductionIntroduction

2.2. Multiresolution AnalysisMultiresolution Analysis

3.3. Hierarchical Displacement MappingHierarchical Displacement Mapping

4.4. Spatial Filtering for Motion DataSpatial Filtering for Motion Data

5.5. ApplicationsApplications

1.1. IntroductionIntroduction

2.2. Multiresolution AnalysisMultiresolution Analysis

3.3. Hierarchical Displacement MappingHierarchical Displacement Mapping

4.4. Spatial Filtering for Motion DataSpatial Filtering for Motion Data

5.5. ApplicationsApplications

Motion Displacement

global (fixed) reference frame

global (fixed) reference frame

TT

ation transformrigid a : T ation transformrigid a : T

)( qpm , )( qpm ,)( qpm , )( qpm ,

33

by zedparameterilocally

RRv)(u,d 33

by zedparameterilocally

RRv)(u,d

Motion Displacement

)exp(

)()(1 vqpquq

vuqp

dmm

,

,,

'

)exp(

)()(1 vqpquq

vuqp

dmm

,

,,

'

global (fixed) reference frame

global (fixed) reference frame

vv

uu)( qpm , )( qpm ,

)( qpm , )( qpm ,

Hierarchical Displacement Mapping

)0(m )0(m

)0(d )0(d

)1(d )1(d

)2(d )2(d

Hierarchical Displacement Mapping

)0(m )0(m

)0(d )0(d

)1(d )1(d

)2(d )2(d

)0(m )0(m

Hierarchical Displacement Mapping

)0(m )0(m

)0(d )0(d

)1(d )1(d

)2(d )2(d

)0(mE )0(mE

Hierarchical Displacement Mapping

)0(m )0(m

)0(d )0(d

)1(d )1(d

)2(d )2(d

)(E 0)0()1( dmm )(E 0)0()1( dmm

A series of successively refined motionsA series of successively refined motions

• Coordinate-independentCoordinate-independent

– measured in a body-fixed coordinate framemeasured in a body-fixed coordinate frame

• UniformityUniformity

– through a local parameterizationthrough a local parameterization

A series of successively refined motionsA series of successively refined motions

• Coordinate-independentCoordinate-independent

– measured in a body-fixed coordinate framemeasured in a body-fixed coordinate frame

• UniformityUniformity

– through a local parameterizationthrough a local parameterization

Hierarchical Displacement Mapping

)))(E(E(E )()()()()( nn dddmm 100 )))(E(E(E )()()()()( nn dddmm 100

Contents

1.1. IntroductionIntroduction

2.2. Multiresolution AnalysisMultiresolution Analysis

3.3. Hierarchical Displacement MappingHierarchical Displacement Mapping

4.4. Spatial Filtering for Motion DataSpatial Filtering for Motion Data

5.5. ApplicationsApplications

1.1. IntroductionIntroduction

2.2. Multiresolution AnalysisMultiresolution Analysis

3.3. Hierarchical Displacement MappingHierarchical Displacement Mapping

4.4. Spatial Filtering for Motion DataSpatial Filtering for Motion Data

5.5. ApplicationsApplications

Spatial Filtering for Orientation Data

Linear shift-invariant (LSI) filterLinear shift-invariant (LSI) filter

• filter mask : filter mask :

• vector-valued signal :vector-valued signal :

Not suitable for unit quaternion dataNot suitable for unit quaternion data

• unit-length constraintsunit-length constraints

Linear shift-invariant (LSI) filterLinear shift-invariant (LSI) filter

• filter mask : filter mask :

• vector-valued signal :vector-valued signal :

Not suitable for unit quaternion dataNot suitable for unit quaternion data

• unit-length constraintsunit-length constraints

kikikiki aaa pppp 0)(F kikikiki aaa pppp 0)(F

R ikk aaaa ),,,,,( 0 R ikk aaaa ),,,,,( 0 3Rp i3Rp i

Previous Work

Re-normalizationRe-normalization

• Azuma and Bishop (‘94)Azuma and Bishop (‘94)

Exploit a local parameterizationExploit a local parameterization

• Lee and Shin (‘96)Lee and Shin (‘96)

• Welch and Bishop (‘97)Welch and Bishop (‘97)

• Fang Fang et al.et al. (‘98) (‘98)

• Hsieh Hsieh et al.et al. (‘98) (‘98)

– lack of crucial filter propertieslack of crucial filter properties

Re-normalizationRe-normalization

• Azuma and Bishop (‘94)Azuma and Bishop (‘94)

Exploit a local parameterizationExploit a local parameterization

• Lee and Shin (‘96)Lee and Shin (‘96)

• Welch and Bishop (‘97)Welch and Bishop (‘97)

• Fang Fang et al.et al. (‘98) (‘98)

• Hsieh Hsieh et al.et al. (‘98) (‘98)

– lack of crucial filter propertieslack of crucial filter properties

Basic Idea

Exploit correspondence in differential spacesExploit correspondence in differential spaces

• Linear motion : Linear motion :

• Angular motion :Angular motion :

Exploit correspondence in differential spacesExploit correspondence in differential spaces

• Linear motion : Linear motion :

• Angular motion :Angular motion : 3)( Sq t 3)( Sq t

3)( Rp t 3)( Rp t

3S3S 3R3R

VelocityVelocity

AccelerationAcceleration '''

'

pω

pω

pq

Transformation

Transformation between linear and angular signalsTransformation between linear and angular signalsTransformation between linear and angular signalsTransformation between linear and angular signals

3S3S 3R3R

2p2p

0p0p

1p1p0q0q

2q2q

1q1q

iiii ppqq

111log iiii ppqq

11

1log

Filter Design

Given: spatial filter Given: spatial filter FF

Output: spatial filter Output: spatial filter HH for orientation data for orientation data

• ““Unitariness” is guaranteedUnitariness” is guaranteed

Given: spatial filter Given: spatial filter FF

Output: spatial filter Output: spatial filter HH for orientation data for orientation data

• ““Unitariness” is guaranteedUnitariness” is guaranteed

iiii ppqq )(exp)( FH iiii ppqq )(exp)( FH

Filter Design

Given: spatial filter Given: spatial filter FF

Output: spatial filter Output: spatial filter HH for orientation data for orientation data

• Local supportLocal support

– #support(#support(HH ) = #support() = #support(FF ))

Given: spatial filter Given: spatial filter FF

Output: spatial filter Output: spatial filter HH for orientation data for orientation data

• Local supportLocal support

– #support(#support(HH ) = #support() = #support(FF ))

1

11logexp

)(exp)(

k

kmmimimi

iiii

b qqq

ppqq FH

1

11logexp

)(exp)(

k

kmmimimi

iiii

b qqq

ppqq FH

Properties of Orientation Filters

Coordinate-invarianceCoordinate-invariance

Time-invarianceTime-invariance

SymmetrySymmetry

Coordinate-invarianceCoordinate-invariance

Time-invarianceTime-invariance

SymmetrySymmetry

ii qqRH,RRH where ii qqRH,RRH where

niinnn

qqSH,SSH where niinnn

qqSH,SSH where

3where Sbabqabaq ,,)H()H( ii3where Sbabqabaq ,,)H()H( ii

Examples (1)

Blurring by binomial masksBlurring by binomial masksBlurring by binomial masksBlurring by binomial masks 1464116

1,,,, 14641

16

1,,,,

OriginalOriginal Angular accelerationAngular accelerationFilteredFiltered

OriginalFilteredOriginalFiltered

Examples (2)

SmoothingSmoothingSmoothingSmoothing 14104116

1 ,,,, 141041

16

1 ,,,,

OriginalOriginal Angular accelerationAngular accelerationFilteredFiltered

OriginalFilteredOriginalFiltered

Examples (3)

High-frequency boostingHigh-frequency boostingHigh-frequency boostingHigh-frequency boosting 14264116

1 ,,,, 142641

16

1 ,,,,

OriginalOriginal Angular accelerationAngular accelerationFilteredFiltered

OriginalFilteredOriginalFiltered

Our scheme vs. Re-normalization

iqiq

1iq 1iq

2iq 2iq

1iq 1iq

2iq 2iq

Re-normalizationiqiq

1iq 1iq

2iq 2iq

1iq 1iq

2iq 2iq

Our scheme

Filtering with an average filterFiltering with an average filterFiltering with an average filterFiltering with an average filter 111115

1,,,, 11111

5

1,,,,

Coordinate Frame-Invariance

mm

)0(

)0(

)1(

)(

mdd

d

n

)0(

)0(

)1(

)(

mdd

d

n

)0(

)0(

)1(

)(

mdd

d

T

n

)0(

)0(

)1(

)(

mdd

d

T

n

mTmTDecomposition Reconstruction

TT

TT

Contents

1.1. IntroductionIntroduction

2.2. Multiresolution AnalysisMultiresolution Analysis

3.3. Hierarchical Displacement MappingHierarchical Displacement Mapping

4.4. Spatial Filtering for Motion DataSpatial Filtering for Motion Data

5.5. ApplicationsApplications

1.1. IntroductionIntroduction

2.2. Multiresolution AnalysisMultiresolution Analysis

3.3. Hierarchical Displacement MappingHierarchical Displacement Mapping

4.4. Spatial Filtering for Motion DataSpatial Filtering for Motion Data

5.5. ApplicationsApplications

Enhancement / Attenuation

Level-wise scaling of coefficientsLevel-wise scaling of coefficientsLevel-wise scaling of coefficientsLevel-wise scaling of coefficients

Motion Blending

Combine multiple motions togetherCombine multiple motions together

• select a base signal and details from different examplesselect a base signal and details from different examples

Combine multiple motions togetherCombine multiple motions together

• select a base signal and details from different examplesselect a base signal and details from different examples

straightwalkingstraightwalking

turningwith a walk

turningwith a walk

straightlimpingstraightlimping

turningwith a limp

turningwith a limp

Motion Blending

Motion Stitching

A simple approachA simple approach

• Estimate velocities at boundaries, thenEstimate velocities at boundaries, then

• Perform -interpolationPerform -interpolation

A simple approachA simple approach

• Estimate velocities at boundaries, thenEstimate velocities at boundaries, then

• Perform -interpolationPerform -interpolation1C1C

Motion Stitching

A simple approachA simple approach

• Estimate velocities at boundaries, thenEstimate velocities at boundaries, then

• Perform -interpolationPerform -interpolation

A simple approachA simple approach

• Estimate velocities at boundaries, thenEstimate velocities at boundaries, then

• Perform -interpolationPerform -interpolation1C1C

Motion Stitching

Difficulties of the simple approachDifficulties of the simple approach

• Hard to estimate velocity robustlyHard to estimate velocity robustly

Difficulties of the simple approachDifficulties of the simple approach

• Hard to estimate velocity robustlyHard to estimate velocity robustly

Motion Stitching

Stitching motion clips seamlesslyStitching motion clips seamlessly

• Merging coefficients level-by-levelMerging coefficients level-by-level

Stitching motion clips seamlesslyStitching motion clips seamlessly

• Merging coefficients level-by-levelMerging coefficients level-by-level

WalkingRunning

Motion Stitching

Stitching motion clips seamlesslyStitching motion clips seamlessly

• Merging coefficients level-by-levelMerging coefficients level-by-level

Stitching motion clips seamlesslyStitching motion clips seamlessly

• Merging coefficients level-by-levelMerging coefficients level-by-level

WalkingRunning

Motion Stitching

Stitching motion clips seamlesslyStitching motion clips seamlessly

• Merging coefficients level-by-levelMerging coefficients level-by-level

Stitching motion clips seamlesslyStitching motion clips seamlessly

• Merging coefficients level-by-levelMerging coefficients level-by-level

stub a toe limp stitching

System DemonstrationSystem DemonstrationSystem DemonstrationSystem Demonstration

Conclusion

Multiresolution motion AnalysisMultiresolution motion Analysis

• Coherency in positions and orientationsCoherency in positions and orientations

• Coordinate-invarianceCoordinate-invariance

Multiresolution motion AnalysisMultiresolution motion Analysis

• Coherency in positions and orientationsCoherency in positions and orientations

• Coordinate-invarianceCoordinate-invariance