Embed Size (px)
DESCRIPTION
3D-Based Reasoning with Blocks, Support, and Stability. Zhaoyin Jia. School of Electrical and Computer Engineering Cornell University. Computer Vision with RGB-D. Pose Recognition J. Shotton et al. 2011; G. Girshick et al. 2013. Activity Detection - PowerPoint PPT Presentation
Citation preview
3D-Based Reasoning with Blocks, Support, and Stability
School of Electrical and Computer EngineeringCornell University
Zhaoyin Jia
2
Computer Vision with RGB-D
Jia, Gallagher, Saxena and Chen
Activity DetectionJ. Sung et al. 2012; H. Koppula et
al. 2013.
Object Recognition K. Lai et al. 2011; A. Janoch et al. 2011
3D Scene Labeling H. Koppula, et al. 2011; N. Silberman et al 2011, 2012.
Pose RecognitionJ. Shotton et al. 2011; G. Girshick et al.
2013.
3
RGB-D Images
Jia, Gallagher, Saxena and Chen
4
3D Reasoning on RGB-D Images Free Space:
objects can be placed in empty spaces.
Physical Stability: one book is
supported by the table and wall.
Foresee Consequences: the camera and the
book will fall if the box moves.
Jia, Gallagher, Saxena and Chen
5
Reasoning with Blocks, Support, & Stability
Input: RGB-D
Jia, Gallagher, Saxena and Chen
Segmentation
6
Reasoning with Blocks, Support, & Stability
Input: RGB-D
Jia, Gallagher, Saxena and Chen
Blocks, Support, and Stability
7
Reasoning with Blocks, Support, & Stability
Input: RGB-D
Jia, Gallagher, Saxena and Chen
Final 3D representation
8
Algorithms
Jia, Gallagher, Saxena and Chen
9
Overview
Jia, Gallagher, Saxena and Chen
3D Block FittingInput Segmentation*
* "Indoor Segmentation and Support Inference from RGBD Images," N. Silberman et al. ECCV, 2012.
Support and Stability
Evaluate Energy
Function
10
3D Block FittingInput Segmentation
Support and Stability
Evaluate Energy
Function
Overview
Jia, Gallagher, Saxena and Chen
3D Block Fitting
11
Single Block Fitting 3D orientated bounding box on depth data Partially observed. Minimum volume may fail * Minimum surface distance (Min-surf)
* "Fast oriented bounding box optimization on the rotation group SO(3, R)," C. Chang et al, ACM Transactions on Graphics, 2011.
Jia, Gallagher, Saxena and Chen
12
Overview
Jia, Gallagher, Saxena and Chen
3D Block FittingInput Segmentation
Support and Stability
Evaluate Energy
Function
Support and Stability
13
Support and Stability
Support Relations
Supporting Area
Stability
14
Support Relation
Jia, Gallagher, Saxena and Chen
Surface On-topSupport
Partial On-topSupport
SideSupport
15 Jia, Gallagher, Saxena and Chen
Surface On-topSupport
Partial On-topSupport
SideSupport
Separate axis is parallel to y
Separate axis is perpendicular to y
16 Jia, Gallagher, Saxena and Chen
Surface On-topSupport
Partial On-topSupport
SideSupport
17
From Support To Stability Supporting Area
Jia, Gallagher, Saxena and Chen
18
From Support To Stability Supporting Area
Stability
Jia, Gallagher, Saxena and Chen
Stable
19
From Support To Stability Supporting Area
Stability
Jia, Gallagher, Saxena and Chen
Stable Unstable
20
Overview
Jia, Gallagher, Saxena and Chen
3D Block FittingInput Segmentation
Support and Stability
Evaluate Energy
Function
Evaluate Energy
Function
21
Reasoning Through an Energy Function
F(S) =
1N
φ(si)i∑ +
1M
ψ (si,sφ)i, φ∑Segmentatio
n Energy Function
Jia, Gallagher, Saxena and Chen
Use Support Relations, Stability, Other Box-based/RGB-D info as features.
RGB-DBetter SegmentationSmaller F(S)
Worse SegmentationLarger F(S)
22
Energy Function: Single Box Potential
Features: minimum surface distance, visibility, single box stability, etc.
Jia, Gallagher, Saxena and Chen
F(S) =
1N
φ(si)i∑ +
1M
ψ (si,sφ)i, φ∑
WorseBox
BetterBox
23
Energy Function: Pairwise Box Potential
Features: box intersection, support, supporting area distance etc.
Jia, Gallagher, Saxena and Chen
F(S ) =
1N
φ(si)i∑ +
1M
ψ (si,sφ)i, φ∑
WorseBoundar
y
BetterBoundar
y
24
Segmentation at one
step
F(S ) =
1N
φ(si)i∑ +
1M
ψ (si,sφ)i, φ∑
Segmentation Energy Function:
1.4
2.3
1.2……
……
……
……
……
Jia, Gallagher, Saxena and Chen
25
Summary
Jia, Gallagher, Saxena and Chen
3D Block FittingInput Segmentation
Support and Stability
Evaluate Energy
Function
26
Experiments
Jia, Gallagher, Saxena and Chen
27
Experiments: Block dataset
Cornell Support Object dataset (SOD) 300 RGB-D images with ground-truth segments and support
relations
NYU-2 RGB-D datasetJia, Gallagher, Saxena and Chen
28
Experiment: Segmentation Results Pixel-wise object segmentation
accuracy:
Jia, Gallagher, Saxena and Chen
Cornell Dataset
NYU Dataset
ECCV-12’ 60.2% 60.1%Ours 70.0% 61.7%
29
Input RGB-D images
Experiment: Segmentation Results
Jia, Gallagher, Saxena and Chen
30
Experiments: Support Inference Neighbor: object is
supported by its neighbors Stability: trim unnecessary
support after reasoning
Jia, Gallagher, Saxena and Chen
Block Dataset
CornellDataset
Neighbor 80.6% 52.9%Stability 91.7% 72.9%
31
Blocks world revisited A. Gupta et all, ECCV, 2010.
Indoor Segmentation & Support
N. Silberman et al. ECCV 2012.Jia, Gallagher, Saxena and Chen
Semantic 3D Labeling H. Koppula et. al. NIPS 2011.
Object Placement Y. Jiang et al. IJRR, 2012.
Color SegmentationD. Hoiem et al. ICCV, 2007;
P. Arbelaez et al. CVPR, 2012.……
32
Conclusion 3D support and stability
Based on box representations
Object segmentation in 3D scene Learning algorithm.
Future work Non-uniform density Semantic classification on blocks Occluded supports
Jia, Gallagher, Saxena and Chen
33
3D-Based Reasoning with Blocks, Support, and StabilityZhaoyin Jia, Andrew Gallagher, Ashutosh Saxena, Tsuhan Chen
Thanks. Questions?
Cornell University
