3DSkeleton-based Human Modeling with Metaballs 18 April 2008 Donghun Kim Robot Vision Lab

3D3D Skeleton-based Human Skeleton-based Human Modeling with MetaballsModeling with Metaballs

18 April 200818 April 2008

DonghunDonghun Kim Kim

Robot Vision LabRobot Vision Lab

ContentsContents

MotivationMotivationWhat is the Metaball?What is the Metaball?How can we visualize Metaballs?How can we visualize Metaballs?3D Skeleton Human Model3D Skeleton Human ModelHuman model using MetaballsHuman model using MetaballsExamplesExamplesFuture WorksFuture Works

MotivationMotivation Motion Analysis of Non-rigid ObjectMotion Analysis of Non-rigid Object Motion Analysis of Articulated Object Motion Analysis of Articulated Object Model based Pose Estimation and Tracking Model based Pose Estimation and Tracking

3D Mesh Data by Visual Hall3D Mesh Data by Visual Hall 3D Skeleton( Curve Skeleton by Thinning, DT, 3D Skeleton( Curve Skeleton by Thinning, DT,

Geometric or General Fields) - on goingGeometric or General Fields) - on going 3D Skeleton-based Motion tracking3D Skeleton-based Motion tracking Metaball Human Model for shape recoveryMetaball Human Model for shape recovery

Motion Analysis by Skeleton-based Motion Motion Analysis by Skeleton-based Motion and Shape Info. (i.e. Manifold learning)and Shape Info. (i.e. Manifold learning)

Implicit SurfacesImplicit Surfaces

Implicit Surfaces: Surfaces which are Implicit Surfaces: Surfaces which are contours(isosurface) through some contours(isosurface) through some scalar field in 3D scalar field in 3D

Different Field function: Metaball, Different Field function: Metaball, Soft Objects, Blobbies Soft Objects, Blobbies

Consider a function f(x,y,z) which define Consider a function f(x,y,z) which define a scalar field in 3D space.a scalar field in 3D space.

Isosurface S is set of points for whichIsosurface S is set of points for which

f(x,y,z) = const.f(x,y,z) = const. It can be thought as an Implicit function It can be thought as an Implicit function

relating x,y and z -> so called implicit relating x,y and z -> so called implicit surface sometimes surface sometimes

What are isosurfaces?What are isosurfaces?

A particularly interesting case A particularly interesting case Use implicit equation of the formUse implicit equation of the form

Gradient can be computed directlyGradient can be computed directly

Soft/blobby objects that blend into each otherSoft/blobby objects that blend into each other

MetaballsMetaballs

11

2

2

N

i i

i

px

r

N

ii

i

i pxpx

rf

14

2

)(2

x: a point in 3d space

N: num. of metaballs

pi : center position of metaballs

ri : Metaball’s own density field

Field FunctionsField Functions

Blobby MoleculesBlobby Molecules

MetaballsMetaballs

Soft ObjectsSoft Objects

2rb)( aerD

rbif

brb

ifb

ra

brif

b

ra

rD

03

)1(2

33

0)3

1(

)( 2

2

2

0

)9

22

9

17

9

41()( 2

2

4

4

6

6

b

r

b

r

b

rarD

r is distance of a point in space to a particular control point

Comparison of Field FunctionsComparison of Field Functions

< Figure from [4]>

ExamplesExamples

< Images from [4]>

Metaballs are cool! Metaballs are cool!

Marching CubesMarching Cubes

Well-known Method for scalr field Well-known Method for scalr field polygonizataion polygonizataion

Sample f(x,y,z) on a cubic latticeSample f(x,y,z) on a cubic latticeFor each cubic cellFor each cubic cell

Estimate where isosurface inetersects Estimate where isosurface inetersects cell edges by linear interpolationcell edges by linear interpolation

Tessellate depending on values of f() at Tessellate depending on values of f() at cell verticescell vertices

Marching CubesMarching Cubes

Algorithm for creating a polygonal surface Algorithm for creating a polygonal surface representation of an isosurface of a 3D representation of an isosurface of a 3D scalar fieldscalar field

Combine simplicity with high speed Combine simplicity with high speed because of using lookup tablesbecause of using lookup tables

ApplicationApplication Reconstruction of a surface from volumetric Reconstruction of a surface from volumetric

datasetsdatasets Creating a 3D contour of a mathematical scalar Creating a 3D contour of a mathematical scalar

fieldfield

Marching CubesMarching Cubes

Each vertex can be either Each vertex can be either ““insideinside”” or or ““outsideoutside””

For each cube cell there are 256 ways for For each cube cell there are 256 ways for isosurface to intersect it can be simplified isosurface to intersect it can be simplified down to 15 uniquedown to 15 unique

Marching CubesMarching Cubes

Example Example cubeindex = 0; if (grid.val[0] < isolevel) cubeindex |= 1; if (grid.val[1] < isolevel) cubeindex |= 2; if (grid.val[2] < isolevel) cubeindex |= 4; if (grid.val[3] < isolevel) cubeindex |= 8; if (grid.val[4] < isolevel) cubeindex |= 16; if (grid.val[5] < isolevel) cubeindex |= 32; if (grid.val[6] < isolevel) cubeindex |= 64; if (grid.val[7] < isolevel) cubeindex |= 128;

< Images from [5]> P = P1 + (isovalue - V1) (P2 - P1) / (V2 - V1)

Marching CubesMarching Cubes

Sampling Grid ResolutionSampling Grid Resolution Smoothness and Processing time to displaySmoothness and Processing time to display

< Images from [5]>

Examples of Marching Examples of Marching CubesCubes

• More Information about Marching Cubes is in [3].

3D Skeleton Human Model3D Skeleton Human Model

Structure( 15 nodes, 14 metaballs)Structure( 15 nodes, 14 metaballs)

0

1

23

4 5

7

68

9 10

1112

13 14

15

Human Model using Human Model using MetaballsMetaballs

Head(1-2-7)Head(1-2-7)

Human Model using Human Model using MetaballsMetaballs

Chest(0-1-2-3-6)Chest(0-1-2-3-6)

Human Model using Human Model using MetaballsMetaballs

Left Arm(1-2-3-8-12)Left Arm(1-2-3-8-12)

Human Model using Human Model using MetaballsMetaballs

Right Arm(1-2-5-11-15)Right Arm(1-2-5-11-15)

Human Model using Human Model using MetaballsMetaballs

Left Hip & Thigh(0-4-9)Left Hip & Thigh(0-4-9)

Human Model using Human Model using MetaballsMetaballs

Right Hip & Thigh(0-5-10)Right Hip & Thigh(0-5-10)

Human Model using Human Model using MetaballsMetaballs

Left Leg(4-9-13)Left Leg(4-9-13)

Human Model using Human Model using MetaballsMetaballs

Right leg(5-10-14)Right leg(5-10-14)

Human Model using Human Model using MetaballsMetaballs

Result (Polygon Surface)Result (Polygon Surface)

Human Model using Human Model using MetaballsMetaballs

Result (Filled surface)Result (Filled surface)

ProgrammingProgramming 3D cloud data and mesh data from Visual 3D cloud data and mesh data from Visual

Hall(using EPVH lib) as the input of 3D skeletonHall(using EPVH lib) as the input of 3D skeleton 3D Human Model Tool based on Jonney3D Human Model Tool based on Jonney’’s 3D s 3D

Human Model Frame (using Ellipsoid, Cylinder, Human Model Frame (using Ellipsoid, Cylinder, Sphere)Sphere)

OpenGL and FLTK GUI based toolOpenGL and FLTK GUI based tool FunctionsFunctions

3D reconstruction by Visual hall3D reconstruction by Visual hall Obtain Multiple Camera-view point images in OpenGL EnvironmentObtain Multiple Camera-view point images in OpenGL Environment Calculate Camera matrix by transforming the OpenGL Calculate Camera matrix by transforming the OpenGL

representation to Physical camera representationrepresentation to Physical camera representation Parameterized 3D Skeleton Human Model Parameterized 3D Skeleton Human Model 3D Skeleton-based Metaball Model3D Skeleton-based Metaball Model Simple model size option (Height considering body ratio, fat-Simple model size option (Height considering body ratio, fat-

thin option) thin option) Image Saving with OpenCV in OpenGLImage Saving with OpenCV in OpenGL

Modeling ToolModeling Tool

Made by Dave Kim and Thanks to Jonney

Modeling ToolModeling Tool

Made by Dave Kim and Thanks to Jonney

Data ComparisonData Comparison

Multi-View Silhouette Images of Multi-View Silhouette Images of Different Human ModelDifferent Human ModelEllipsoid, Cylinder and Sphere vs. Ellipsoid, Cylinder and Sphere vs.

MetaballsMetaballs3D cloud points and triangle meshes 3D cloud points and triangle meshes

by Visual hallby Visual hall

Example (I)Example (I)

Example (I)Example (I)

Example (II)Example (II)

Metaball Human ModelMetaball Human Model

Example (II)Example (II)

Result by Visual HallResult by Visual Hall

Future WorkFuture Work

3D Curve skeleton Algorithm[2] 3D Curve skeleton Algorithm[2] DT (fast) + General Field (robust)DT (fast) + General Field (robust)

3D skeleton human model fitting to 3D skeleton human model fitting to Curve Skeleton -> Pose EstimationCurve Skeleton -> Pose Estimation

Shape recovery using the information Shape recovery using the information from the skeletonizationfrom the skeletonization

Motion AnalysisMotion Analysis

ReferenceReference1.1. Clement Menier, Bruno Raffin, Clement Menier, Bruno Raffin, ““3D Skeleton-based Pose 3D Skeleton-based Pose

Recovery,Recovery,”” The 3 The 3rdrd international Symposimum on 3D Data international Symposimum on 3D Data Processing, Visualization and Transmission, 2006Processing, Visualization and Transmission, 2006

2.2. Nicu D. Cormea etc at al., Nicu D. Cormea etc at al., ““Curve-Skeleton Application,Curve-Skeleton Application,”” IEEE Visualization 2005IEEE Visualization 2005

3.3. T. S. Newman, H. Yi, T. S. Newman, H. Yi, ““A survey of the marching cubes A survey of the marching cubes algorithm,algorithm,”” Computers and Graphics, 2006 Computers and Graphics, 2006

4.4. Implicit Surfaces written by Paul Bourke, June 1997, Implicit Surfaces written by Paul Bourke, June 1997, http://local.wasp.uwa.edu.au/~pbourke/modelling_renderihttp://local.wasp.uwa.edu.au/~pbourke/modelling_rendering/implicitsurf/ng/implicitsurf/

5.5. Polygonising A Scalar Field written by Paul Bourke, May Polygonising A Scalar Field written by Paul Bourke, May 1994, 1994, http://local.wasp.uwa.edu.au/~pbourke/geometry/polygohttp://local.wasp.uwa.edu.au/~pbourke/geometry/polygonisenise