Embed Size (px)
Citation preview
Structure Recovery by Part Assembly
Chao-Hui Shen1 Hongbo Fu2 Kang Chen1 Shi-Min Hu1
1Tsinghua University 2City University of Hong Kong
Background
• Consumer level scanning devices• Capture both RGB and depth• Reconstruction is challenging
– Low resolution– Noise– Missing data– …
Example-based Scan Completion
• Global-to-local and top-down [Kraevoy and Sheffer 2005; Pauly et al. 2005]
• Rely on the availability of suitable template model• However …
No suitable model!shape retrieval
Assembly-based 3D Modeling
• Data-drive suggestion and interaction [Chaudhuri and Koltun 2010; Chaudhuri et al. 2011]– Retrieve suitable parts to match user intent– Aim to support open-ended 3D modeling– Quite different goal from ours
• Automatic shape synthesis by part composition [Kalogerakis et al. 2012; Jain et al. 2012; Xu et al. 2012] – Result in database that grows exponentially– Significantly enlarge the existing database– But make storage and retrieval challenging
Our solution: Recover the Structure by Part Assembly• Structure recovery instead of geometry reconstruction• Do NOT prepare a large database• Retrieve and assemble suitable parts on the fly
Problem Setup
Input
Point cloud + Image (Single view)
Pre-segmented Repository Models (Parts + Labels)
……
Goal: Recover high-level structure
Assembly close to geometry
Output
……
Session: Acquiring and Synthesizing Indoor Scenes
An Interactive Approach to Semantic Modeling of Indoor Scenes with an RGBD Camera [Shao et al. 2012]
A Search-Classify Approach for Cluttered Indoor Scene Understanding [Nan et al. 2012]
Acquiring 3D Indoor Environments with Variability and Repetition [Kim et al. 2012]
• Directly searching is computationally prohibitive
• Need a quick way to explore meaningful structures guided by:– Spatial layout of the parts in the repository models– Acquired data
Observations
Observations
• Complementary characteristics of point cloud & image
3D, more accurate cues for geometry & structure
Incomplete and noisy
Lack depth information
Capture the complete object
Algorithm Overview
Candidate Parts Selection Structure Composition Part Conjoining
……
Algorithm Overview
Candidate Parts Selection Structure Composition Part Conjoining
……
Candidate Parts Selection
• Goal: select a small set of candidates for each category • Achieved by retrieving parts that fit well some regions
Straightforward Solution
• Search for the best-fit parts over the entire domain– Disregards the semantics associated with each part and
the interaction between different parts
Unlikely to produce good results!X X X
X XX X X
Key Fact
• Man-made objects lie in a low dimensional space– Defined with respect to the relative sizes and positions of
shape parts [Ovsjanikov et al. 2011]
• Employ 3D repository model as a global context– Globally align the models with the input scan first
Search in a 3D offset window around the part
Part Matching Scheme
(part contour)
(2D field)
Geometric fidelity score
Geometric contribution score
3D 2Dedgemap
Total matching score
¿{𝑪𝒑 (𝒊 , 𝒋 )=𝟏 }
3D offset window
Candidate Parts
• Select top K parts with highest score for each category
Seat
Back
Arm
Front leg
……
……
……
……
……
…… …… …… …… …… ……
Algorithm Overview
Candidate Parts Selection Structure Composition Part Conjoining
……
Structure Composition
• Goal: compose the underlying structure by identifying a subset of candidate parts
Constraints for Promising Compositions
Geometric fidelity Proximity Overlap
having high score no isolated parts minimized intersection
Search and Evaluate
• Search for promising compositions under constraints
• Globally Evaluate the compositions
average geometry fidelity of parts total geometry fidelity
total geometry contribution
……optimal composition
Algorithm Overview
Candidate Parts Selection Structure Composition Part Conjoining
……
Part Conjoining
• Problem: the parts are loosely placed together• Goal: generate a consistent & complete model
Identification of Contact Points
• Refer to their parent models [Jain et al. 2012]
Matching of Contact Points
• Greedily match nearby contact points• Generate auxiliary contact points when necessary
auxiliary contact points
𝒑𝒎𝒊𝒌
𝒑𝒏𝒋𝒌
i j
identity scale
Global Optimization
transformed contact points
• Adjust the sizes {} and positions {} of parts• Make matched point as close as possible• Contact enforcement
• Shape preserving
• Global optimization
Results: Chairs
• 70 repository models, 11 part categories
Results: Tables
• 61 repository models, 4 part categories
Results: Bicycles
• 38 repository models, 9 part categories
Results: Airplanes
• 70 repository models, 6 part categories
Results: Creating New Structures
Results: Impact of Dataset
input data
Randomly picking some repository models
Summary
• A bottom-up structure recovery approach– Effectively reuse limited repository models– Automatically compose new structure– Handle single-view inputs by the Kinect system
• Future work– Multi-view inputs– Include style/functional constraints– Recover Indoor scenes
Thank you!
Project Page: http://cg.cs.tsinghua.edu.cn/StructureRecovery
