INFORMATIKINFORMATIK
Hendrik LenschHendrik Lensch
Hardware-AcceleratedHardware-AcceleratedSilhouette MatchingSilhouette Matching
Hendrik Lensch, Wolfgang Heidrich,Hendrik Lensch, Wolfgang Heidrich, and Hans-Peter Seideland Hans-Peter Seidel
Max-Planck-Institut fMax-Planck-Institut für Informatik,ür Informatik,Saarbrücken (Germany)Saarbrücken (Germany)
Hendrik Lensch, Wolfgang Heidrich,Hendrik Lensch, Wolfgang Heidrich, and Hans-Peter Seideland Hans-Peter Seidel
Max-Planck-Institut fMax-Planck-Institut für Informatik,ür Informatik,Saarbrücken (Germany)Saarbrücken (Germany)
INFORMATIKINFORMATIK
Hendrik LenschHendrik Lensch
OverviewOverview
• MotivationMotivation• Comparing SilhouettesComparing Silhouettes• Stitching and Combining TexturesStitching and Combining Textures• Results and ConclusionsResults and Conclusions
• MotivationMotivation• Comparing SilhouettesComparing Silhouettes• Stitching and Combining TexturesStitching and Combining Textures• Results and ConclusionsResults and Conclusions
INFORMATIKINFORMATIK
Hendrik LenschHendrik Lensch
Acquiring Real World ModelsAcquiring Real World Models
GeometryGeometry• 3D scanner3D scanner
GeometryGeometry• 3D scanner3D scanner
Texture dataTexture data• digital cameradigital camera
Texture dataTexture data• digital cameradigital camera
single sensor vs. multiple sensorssingle sensor vs. multiple sensors
INFORMATIKINFORMATIK
Hendrik LenschHendrik Lensch
3D – 2D Registration3D – 2D Registration
Find the camera setting for each 2D image.Find the camera setting for each 2D image.Find the camera setting for each 2D image.Find the camera setting for each 2D image.
INFORMATIKINFORMATIK
Hendrik LenschHendrik Lensch
TransformationsTransformations• to camera coordinates (extrinsic):to camera coordinates (extrinsic):
• to 2D image space (intrinsic):to 2D image space (intrinsic):
determine determine R, tR, t and and ff (6+1 dimensions) (6+1 dimensions)
TransformationsTransformations• to camera coordinates (extrinsic):to camera coordinates (extrinsic):
• to 2D image space (intrinsic):to 2D image space (intrinsic):
determine determine R, tR, t and and ff (6+1 dimensions) (6+1 dimensions)
tgxR
z
y
x
w
c
c
c
Camera ModelCamera Model
c
c
c y
x
z
f
v
u
v
u1
0
0
INFORMATIKINFORMATIK
Hendrik LenschHendrik Lensch
Similarity MeasureSimilarity Measure
Which features to investigate?Which features to investigate?• no color information on the modelno color information on the model• correspondence of geometric features hard to findcorrespondence of geometric features hard to find
Which features to investigate?Which features to investigate?• no color information on the modelno color information on the model• correspondence of geometric features hard to findcorrespondence of geometric features hard to find
INFORMATIKINFORMATIK
Hendrik LenschHendrik Lensch
Similarity MeasureSimilarity Measure
Compare silhouettes Compare silhouettes [Etienne de Silhouette 1709-1767][Etienne de Silhouette 1709-1767]
• model: render monochromemodel: render monochrome• photo: automatic histogram-based segmentationphoto: automatic histogram-based segmentation
Compare silhouettes Compare silhouettes [Etienne de Silhouette 1709-1767][Etienne de Silhouette 1709-1767]
• model: render monochromemodel: render monochrome• photo: automatic histogram-based segmentationphoto: automatic histogram-based segmentation
INFORMATIKINFORMATIK
Hendrik LenschHendrik Lensch
Similarity MeasureSimilarity Measure
Compare silhouettes Compare silhouettes [Etienne de Silhouette 1709-1767][Etienne de Silhouette 1709-1767]
• model: render monochromemodel: render monochrome• photo: automatic histogram-based segmentationphoto: automatic histogram-based segmentation
Compare silhouettes Compare silhouettes [Etienne de Silhouette 1709-1767][Etienne de Silhouette 1709-1767]
• model: render monochromemodel: render monochrome• photo: automatic histogram-based segmentationphoto: automatic histogram-based segmentation
INFORMATIKINFORMATIK
Hendrik LenschHendrik Lensch
Distance Measure for Distance Measure for SilhouettesSilhouettes
Point-to-outline Point-to-outline distancesdistances
• slow because points on slow because points on the outline must be the outline must be determineddetermined
• speedup by distance speedup by distance mapsmaps
Point-to-outline Point-to-outline distancesdistances
• slow because points on slow because points on the outline must be the outline must be determineddetermined
• speedup by distance speedup by distance mapsmaps
INFORMATIKINFORMATIK
Hendrik LenschHendrik Lensch
11
00
intensityintensity
xx
differencedifference
11
00xx
Pixel-based Pixel-based Distance MeasureDistance Measure
Count the number of Count the number of pixels covered by pixels covered by just one silhouette.just one silhouette.• XOR the imagesXOR the images
• compute histogram compute histogram (hardware)(hardware)
• gives linear response to gives linear response to the displacementthe displacement
Count the number of Count the number of pixels covered by pixels covered by just one silhouette.just one silhouette.• XOR the imagesXOR the images
• compute histogram compute histogram (hardware)(hardware)
• gives linear response to gives linear response to the displacementthe displacement
displacementdisplacement
INFORMATIKINFORMATIK
Hendrik LenschHendrik Lensch
1111
00
intensityintensity
xx
11
00
differencedifference
xx
Pixel-based Pixel-based Distance MeasureDistance Measure
Count the number of Count the number of pixels covered by pixels covered by just one silhouette.just one silhouette.• XOR the imagesXOR the images
• compute histogram compute histogram (hardware)(hardware)
• gives linear response to gives linear response to the displacementthe displacement
Count the number of Count the number of pixels covered by pixels covered by just one silhouette.just one silhouette.• XOR the imagesXOR the images
• compute histogram compute histogram (hardware)(hardware)
• gives linear response to gives linear response to the displacementthe displacement
displacementdisplacement
INFORMATIKINFORMATIK
Hendrik LenschHendrik Lensch
Approximation ofApproximation ofSquared DistancesSquared Distances
Use smooth transitionsUse smooth transitions• blur imagesblur images
• integrate squared integrate squared differencesdifferences
• faster convergencefaster convergence• reduced variance reduced variance • higher evaluation costhigher evaluation cost
Use smooth transitionsUse smooth transitions• blur imagesblur images
• integrate squared integrate squared differencesdifferences
• faster convergencefaster convergence• reduced variance reduced variance • higher evaluation costhigher evaluation cost
INFORMATIKINFORMATIK
Hendrik LenschHendrik Lensch
Approximation ofApproximation ofSquared DistancesSquared Distances
Use smooth transitionsUse smooth transitions• blur imagesblur images
• integrate squared integrate squared differencesdifferences
• faster convergencefaster convergence• reduced variance reduced variance • higher evaluation costhigher evaluation cost
Use smooth transitionsUse smooth transitions• blur imagesblur images
• integrate squared integrate squared differencesdifferences
• faster convergencefaster convergence• reduced variance reduced variance • higher evaluation costhigher evaluation cost
1111
00
intensityintensity
xx
1111
00
intensityintensity
xx
11
00
differencedifference
xx
differencedifference
11
00xx
filtersizefiltersize
INFORMATIKINFORMATIK
Hendrik LenschHendrik Lensch
Non-linear Optimization Non-linear Optimization
Downhill Simplex Method [Press 1992] Downhill Simplex Method [Press 1992] • works for works for NN dimensions dimensions• no derivatives no derivatives • easy to controleasy to control
Downhill Simplex Method [Press 1992] Downhill Simplex Method [Press 1992] • works for works for NN dimensions dimensions• no derivatives no derivatives • easy to controleasy to control
INFORMATIKINFORMATIK
Hendrik LenschHendrik Lensch
Simplex Method in 3D Simplex Method in 3D
original simplexoriginal simplexoriginal simplexoriginal simplex
reflection and/orreflection and/orexpansionexpansionreflection and/orreflection and/orexpansionexpansion
shrinkingshrinkingshrinkingshrinking
random random perturbationperturbationrandom random perturbationperturbation
INFORMATIKINFORMATIK
Hendrik LenschHendrik Lensch
Hierarchical OptimizationHierarchical Optimization
• optimize on low resolution firstoptimize on low resolution first• restart optimization to avoid local minimarestart optimization to avoid local minima• switch to higher resolutionswitch to higher resolution• mesh resolution can be adaptedmesh resolution can be adapted
• optimize on low resolution firstoptimize on low resolution first• restart optimization to avoid local minimarestart optimization to avoid local minima• switch to higher resolutionswitch to higher resolution• mesh resolution can be adaptedmesh resolution can be adapted
INFORMATIKINFORMATIK
Hendrik LenschHendrik Lensch
Starting Point GenerationStarting Point Generation
• set camera distance set camera distance ttzz depending on object size depending on object size
• setset ttxx and and ttyy to zero to zero• select 48 sample rotationsselect 48 sample rotations• run optimization for each of the samplesrun optimization for each of the samples
(40 evaluations)(40 evaluations)• select top 5 resultsselect top 5 results• restart optimization (200 evaluations)restart optimization (200 evaluations)• take best result as starting pointtake best result as starting point
• set camera distance set camera distance ttzz depending on object size depending on object size
• setset ttxx and and ttyy to zero to zero• select 48 sample rotationsselect 48 sample rotations• run optimization for each of the samplesrun optimization for each of the samples
(40 evaluations)(40 evaluations)• select top 5 resultsselect top 5 results• restart optimization (200 evaluations)restart optimization (200 evaluations)• take best result as starting pointtake best result as starting point
INFORMATIKINFORMATIK
Hendrik LenschHendrik Lensch
Texture StitchingTexture Stitching
projective texture mappingprojective texture mapping assign one image to each triangle assign one image to each triangle • triangle visible in image? (test every vertex)triangle visible in image? (test every vertex)• select best viewing angleselect best viewing angle• discard data near depth discontinuities discard data near depth discontinuities
projective texture mappingprojective texture mapping assign one image to each triangle assign one image to each triangle • triangle visible in image? (test every vertex)triangle visible in image? (test every vertex)• select best viewing angleselect best viewing angle• discard data near depth discontinuities discard data near depth discontinuities
INFORMATIKINFORMATIK
Hendrik LenschHendrik Lensch
Blending Across Blending Across Assignment BordersAssignment Borders
• find border verticesfind border vertices
• release all triangles release all triangles around them around them
• assign boundary assign boundary vertices to best regionvertices to best region
• assign alpha-values for assign alpha-values for each region each region
– 1 to vertices 1 to vertices included in the regionincluded in the region
– 0 to all others.0 to all others.
• find border verticesfind border vertices
• release all triangles release all triangles around them around them
• assign boundary assign boundary vertices to best regionvertices to best region
• assign alpha-values for assign alpha-values for each region each region
– 1 to vertices 1 to vertices included in the regionincluded in the region
– 0 to all others.0 to all others.
INFORMATIKINFORMATIK
Hendrik LenschHendrik Lensch
Entire TextureEntire Texture
INFORMATIKINFORMATIK
Hendrik LenschHendrik Lensch
Results and ConclusionsResults and Conclusions
Problems solved:Problems solved:• automatic texture registration (automatic texture registration (R, t, fR, t, f))• view-independent texture stitchingview-independent texture stitching• blending across assignment boundariesblending across assignment boundaries• rough manual alignment helps (speedup, failures) rough manual alignment helps (speedup, failures)
Further problems:Further problems:• extract purely diffuse part of textureextract purely diffuse part of texture• generate texture where data is missinggenerate texture where data is missing
Problems solved:Problems solved:• automatic texture registration (automatic texture registration (R, t, fR, t, f))• view-independent texture stitchingview-independent texture stitching• blending across assignment boundariesblending across assignment boundaries• rough manual alignment helps (speedup, failures) rough manual alignment helps (speedup, failures)
Further problems:Further problems:• extract purely diffuse part of textureextract purely diffuse part of texture• generate texture where data is missinggenerate texture where data is missing
INFORMATIKINFORMATIK
Hendrik LenschHendrik Lensch
Questions?Questions?
visit us at visit us at
www.mpi-sb.mpg.dewww.mpi-sb.mpg.de
visit us at visit us at
www.mpi-sb.mpg.dewww.mpi-sb.mpg.de