3D Shape 3D Shape Registration using Registration using Regularized Medial Regularized Medial

ScaffoldsScaffolds

Ming-Ching ChangMing-Ching ChangFrederic F. Leymarie Frederic F. Leymarie

Benjamin B. KimiaBenjamin B. Kimia

LEMS, Division of Engineering, Brown University

3DPVT 2004 Thessaloniki, GreeceSep. 6-9, 2004

OutlineOutline Registration BackgroundRegistration Background Medial Scaffold: Representation for Medial Scaffold: Representation for

3D Shapes3D Shapes Graduated Assignment Graph Graduated Assignment Graph

MatchingMatching ResultsResults ConclusionsConclusions

Registration: Defining Registration: Defining CorrespondenceCorrespondence

Local RegistrationLocal RegistrationInitial position given. ICP and it’s improvementsInitial position given. ICP and it’s improvementsSurvey: [Campbell & Flynn CVIU’01], [3DIM’03]Survey: [Campbell & Flynn CVIU’01], [3DIM’03]

Global Registration Global Registration Skeleton-based, Surface-feature basedSkeleton-based, Surface-feature based

Fundamental for processing scanned objects, modeling, Fundamental for processing scanned objects, modeling, matching, recognition, medical applications, etc.matching, recognition, medical applications, etc.

More difficult.Main focus of this talk.

Mesh with 20K points

2K points

Local Registration: Iterative Closest Local Registration: Iterative Closest Points (ICP)Points (ICP)

[Besl & McKay PAMI’92][Besl & McKay PAMI’92] Needs a good Needs a good initial alignmentinitial alignment Local search problemsLocal search problems

Sensitive to local minimum, noiseSensitive to local minimum, noise May converge slowlyMay converge slowly Lack of surface representationLack of surface representation

Improvements:Improvements: [Chen & Mendioni] [Chen & Mendioni] accuracyaccuracy: :

match closest point on the match closest point on the projected planeprojected plane

Use color, Use color, non-rigidnon-rigid match to get match to get better better convergenceconvergence, etc., etc.

Iter. 1

Iter. 5

Global RegistrationGlobal Registration Surface featured basedSurface featured based

[Wyngaerd & Van Gool CVIU’02]: [Wyngaerd & Van Gool CVIU’02]: bitangent bitangent curve pairscurve pairs as surface landmarksas surface landmarks

[Allen[Allen et. al. et. al.’03]: ’03]: straight linesstraight lines as features as featuresin aligning architectural datasetin aligning architectural dataset

Skeletal graph basedSkeletal graph based [Brennecke & Isenberg ’04]: [Brennecke & Isenberg ’04]:

Internal skeletal graphInternal skeletal graph of a of a closedclosed surface surface mesh, using an edge collapse algorithmmesh, using an edge collapse algorithm

match match largest common subgraphlargest common subgraph [Sundar [Sundar et. al. et. al. ’03]: ’03]: Skeletal Skeletal treetree from thinning from thinning

voxels via voxels via a distance transform, coarse-to-fine matchinga distance transform, coarse-to-fine matching1. Skeletons over-simplified 2. Graph topology not handled well

Proposed: Match the Medial Proposed: Match the Medial ScaffoldScaffold

Medial ScaffoldMedial Scaffold: medial structure in the form of a 3D hypergraph: medial structure in the form of a 3D hypergraph

Medial ScaffoldMedial Scaffold Blum’s medial axis (grassfire), wave Blum’s medial axis (grassfire), wave

propagationpropagation

3D: Five types of points 3D: Five types of points [Giblin & Kimia PAMI’04][Giblin & Kimia PAMI’04]:: SheetSheet: A: A11

22

LinksLinks: A: A113 3 (Axial), A(Axial), A3 3 (Rib)(Rib)

NodesNodes: A: A1144, A, A11AA33

AAkknn: contact at : contact at nn distinct points, each with distinct points, each with

k+1k+1 degree of contact degree of contact

A3

A13

Shock

A3

A12

A13

Medial Structure Medial Structure RegularizationRegularization

Medial Axis is sensitive to noise & perturbations.Medial Axis is sensitive to noise & perturbations. TransitionsTransitions: sudden changes in topology : sudden changes in topology 2D examples:2D examples:

The growth of an axis with small perturbations (A1A3)

The swapping of MA branches (A1

4)

Smoothing/medial branch pruning

Pruning:

Seven Types of Transitions Seven Types of Transitions in 3Din 3D

A1A3

-I

A15

A14

A5

A1A3-II

A12A3-I

A12A3-II

[Giblin & Kimia ECCV’02]

Scaffold Scaffold RegularizationRegularization

Transition removal, i.e. remove Transition removal, i.e. remove topological instabilitytopological instability SmoothingSmoothing

Green: A1A3 nodes, Pink: A14 nodesBlue: A3 links, Red:

A14 links

A1A3-I

A5

A15

A12A3-I

[Leymarie et. al. ICPR’04]

Match Medial Scaffolds by Match Medial Scaffolds by Graph MatchingGraph Matching

IntractabilityIntractability Weighted graph matching: Weighted graph matching: NPNP-hard-hard One special case: Largest common subgraph: One special case: Largest common subgraph: NPNP--

completecomplete Only “good” Only “good” sub-optimalsub-optimal solutions can be found solutions can be found

Graduated Assignment Graduated Assignment [Gold & Rangarajan [Gold & Rangarajan PAMI’96]PAMI’96] [Sharvit [Sharvit et. al. et. al. JVCIR’98]JVCIR’98] index 25-shape database index 25-shape database

by matching 2D by matching 2D shock graphsshock graphs

3D 3D hypergraphhypergraph matching: matching: Additional dimensionAdditional dimension Generally not a Generally not a treetree, might have isolated , might have isolated loopsloops No inside/outside: non-closed surfaces or surface No inside/outside: non-closed surfaces or surface

patchespatches

QuadraticQuadratic weighted graph weighted graph matchingmatching

G, G: 2 undirected graphsG, G: 2 undirected graphsI: # of nodes in G, I: # of nodes in GI: # of nodes in G, I: # of nodes in G{G{Gii}, {G}, {Gii} nodes} nodes{G{Gijij}, {G}, {Gijij} edges: } edges: adjacency matricesadjacency matrices of of

graphsgraphs

The The match matrixmatch matrix MMiiii = 1 if node i in G corresponds to node i in = 1 if node i in G corresponds to node i in

G, G, = 0 otherwise= 0 otherwise

a

b cz

G: o

p

qGG:

Graduated AssignmentGraduated Assignment

Then objective function to maximize over the Then objective function to maximize over the space of space of MM is: is:

LLiijjiijj: link similarity between G: link similarity between Gijij and G and Gijij

NNiiii: node similarity between G: node similarity between Gii and G and Gii

Cost of matching Gij to Gij.If the nodes match, how similar the links are.

Cost of matching Gi to Gi

Modified Graduated Assignment Modified Graduated Assignment for 3D Medial Scaffold Matchingfor 3D Medial Scaffold Matching

Node cost: (radius)

Link cost: (length)

Sheet (hyperlink) cost: (corner angle)

α, β: weights

Results: Results: SheepSheep

Sheep 20K points, after surface reconstruction

Sheep 1-20K Full Scaffold

The scaffold matching is good enough that ICP is not required.

Result of Scaffold Graph Result of Scaffold Graph MatchingMatching

Colors to represent correct link matches;grays to represent miss matches.

Two scans of an object at the same resolution (20K points):

Results: David HeadResults: David Head20K

30K

Two sub-samples from the ground truth (42350 points)

Matching ResultsMatching ResultsScaffold matching result Scaffold matching + ICP

Validation against the ground truth: (object dimension = 69x69x76) average sq dist 3.129372 average sq dist 0.000005

Partial Shape Matching: Partial Shape Matching: Sheep with the rear portion Sheep with the rear portion

cut offcut off

Sheep 1-20K with the rear portion cutSheep 1-20K scaffold

Partial Shape Matching Partial Shape Matching ResultResult

Non-closed Surface: Non-closed Surface: Archaeological PotArchaeological Pot

Two scans of the outside surface of a pot (50K and 40K). The inner surface of the pot is missing.

Alignment by Scaffold Alignment by Scaffold MatchingMatching

The scaffold matching result

Final Registration after Final Registration after ICPICP

Two Possible Reasons for Two Possible Reasons for Incorrect MatchesIncorrect Matches

Graduated assignment matching is not optimal.Graduated assignment matching is not optimal.

Typically this does not affect the overall registration if a sufficient number of nodes are correctly assigned.

MIS-MATCH!!

Pot sherd 1 (50K) Pot sherd 2 (10K)

1. Only 8 shock vertices to match2. Transitions not completely handled Result of shock matching

Reasons for Incorrect Matches Reasons for Incorrect Matches (cont’d)(cont’d)

Medial structure transitions are not completely handled.Medial structure transitions are not completely handled.

MIS-MATCH!!

Benefits of 3D Medial Benefits of 3D Medial ScaffoldsScaffolds

A global A global hierarchicalhierarchical structure is structure is built-in.built-in.

ScaleScale is represented. is represented. Salient features are captured:Salient features are captured:

Generalized axesGeneralized axes of elongated objects of elongated objects curvature extremacurvature extrema and and ridgesridges

The medial representation is The medial representation is completecomplete. . ReconstructionReconstruction of the shape of the shape is always possible.is always possible.

RobustRobust after regularization. after regularization. Easy to handle shape Easy to handle shape

deformationsdeformations.. Data from Cyberware Inc.

ConclusionsConclusions

Take input as point clouds or partial meshes.Take input as point clouds or partial meshes. Robust to noise. Invariant under different resolutions Robust to noise. Invariant under different resolutions

& acquisition conditions.& acquisition conditions.

Can be Can be graphsgraphs with loops (not a with loops (not a treetree).). Contains Contains sheetssheets, , linkslinks, , nodesnodes. Not over-simplified.. Not over-simplified. Carefully Regularized.Carefully Regularized.

Global Registration by Matching Medial ScaffoldsGlobal Registration by Matching Medial Scaffolds

- Skeleton:

Nearly-optimal.Nearly-optimal. Can be improved to doCan be improved to do finefine registration. registration. Can be extended to register Can be extended to register non-rigidnon-rigid objects. objects. Can be extended to do Can be extended to do recognitionrecognition..

- Match:

Thank YouThank You

This material is based upon work supported This material is based upon work supported by the National Science Foundation under by the National Science Foundation under Grants 0205477 and 0083231.Grants 0205477 and 0083231.

AcknowledgementsAcknowledgements