Slide 1

Patch Based Synthesis for Single Depth Image Super-Resolution (ECCV 2012) Oisin Mac Aodha, Neill Campbell, Arun Nair and Gabriel J. Brostow Presented By: Itzik Ben ShabatJanuary 2014

Slide 2

Contents Problem & Motivation SR General Overview Related Work The Proposed Method Results Qualitative Quantitative Future Work Paper review

Slide 3

Problem & Motivation How do we convert a Low Resolution (LR) image to High Resolution (HR) ? Get a better camera (Sensor)

Slide 4

Problem & Motivation Cant get a better camera? Super Resolve the image! (SR)

Slide 5

Problem & Motivation Now do it in RGB-D ! ! ! PMD CamCube - 200X200 PointGrey BumbleBee 2 - 640x480 at 48fps MS Kinect - 640x480

Slide 6

SR General Overview Common approaches: Take multiple LR images from different angles and reconstruct the additional information (requires multiple images)

Slide 7

SR General Overview Common approaches: Use a LR to HR database (requires a database) Focus on this approach

Slide 8

Related Work Intensity Images EbSR [15] - Freeman, W.T., Liu, C.: Example-based super resolution. In: Advances in Markov Random Fields for Vision and Image Processing. MIT Press (2011) Similar: Filter input Normalized Patch matching Solving minimum energy problem (using BP) Different Not Designed for RGB-D images Matching HR and interpolated LR patches Create LR- HR database Perform bicubic interpolation on input image Desaturate and High- pass filtered Normalize contrast Solve patch minimum energy problem using BP Add back low frequency and color

Slide 9

Related Work EbSR looking closer Ground trouth EbSR Output

Slide 10

Intensity Images ScSr [17] - Yang, J., Wright, J., Huang, T.S., Ma, Y.: Image super-resolution via sparse representation. IEEE Transactions on Image Processing (2010) Similar: Use patches and minimization problem Different: Not designed for RGB-D images Two dictionaries Database structure (sparse representation) Solves 2 minimization problems separately global and local No noise reduction implementation Learn 2 dictionaries (LR- HR) For each input patch - Find LR representation Apply representation to HR pairs Solve global optimization problem Related Work

Slide 11

Depth + Intensity Hybrids Cross Bilateral [1] - Yang, Q., Yang, R., Davis, J., Nister, D.: Spatial-depth super resolution for range images. In: CVPR. (2007) Similar: Specific for RGB-D Use bilateral filter Different: Requires additional input (destination resolution image) Doesnt use patches Solves a fusion problem

Slide 12

Construct Multi- Resolution MRF grid Formulate optimization problem Bilinear filter for y initial guess Solve iteratively Related Work Depth + Intensity Hybrids MRF SR [25]- Diebel, J., Thrun, S.: An application of markov random fields to range sensing. In: NIPS. (2005) Similar: Uses MRF Different: Uses multi-resolution MRF Requires additional input (destination resolution image) Doesnt use patches Solves a fusion problem

Slide 13

The Proposed Method Challenges Construct database Noise Flying pixels at discontinuities Wrong depths for specular or dark materials Edges jarring artifacts (different than rgb image)

Slide 14

The Proposed Method - Overview Construct Database Filter Input Generate Patches of input image Find Match Candidates Solve Minimum Energy Problem Reconstruct ImagePost Processing Filtering

Slide 15

The Proposed Method Database Constructing the database Less sources for database construction than rgb images Considered synthetic Vs. Real datasets Database uses 30 scenes of 800x800 (scenes flipped left to right) 5.3 million patches Pruning to remove redundant patches (planar surfaces) SyntheticLaser Scan

Slide 16

The Proposed Method - Overview Construct Database Filter Input Generate Patches of input image Find Match Candidates Solve Minimum Energy Problem Reconstruct ImagePost Processing Filtering

Slide 17

The Proposed Method - Filtering Noise Reduction Assumption High frequency=noise A. Bilateral filter on input patches before patch normalization Edge preserving Noise reducing Nonlinear Weighted average of intensity values from nearby pixels *Used in Adobe Photoshop Blur function B. Bicubic filter on database HR patches before down-sampling

Slide 18

The Proposed Method - Filtering Input After Bilateral filtering

Slide 19

The Proposed Method - Filtering Noise Reduction Pro Cleaner image for patching Con Some data is lost

Slide 20

The Proposed Method - Overview Construct Database Filter Input Generate Patches of input image Find Match Candidates Solve Minimum Energy Problem Reconstruct ImagePost Processing Filtering

Slide 21

The Proposed Method Matching Input Depth Image

Slide 22

N non overlapping low resolution input patches x i For each x i we wish to find its corresponding high resolution y i Patches are normalized The Proposed Method Matching Input Depth Image

Slide 23

The Proposed Method Matching High Resolution Database

Slide 24

Output Depth Image The Proposed Method Matching Input Depth Image High Resolution Database

Slide 25

Output Depth Image High Resolution Database The Proposed Method Matching Input Depth Image

Slide 26

Output Depth Image High Resolution Database The Proposed Method Matching

Slide 27

Output Depth Image The Proposed Method Matching Input Depth Image High Resolution Database

Slide 28

Output Depth Image High Resolution Database The Proposed Method Matching Input Depth Image

Slide 29

Output Depth Image The Proposed Method Matching Input Depth Image High Resolution Database

Slide 30

Output Depth Image The Proposed Method Matching Input Depth Image High Resolution Database

Slide 31

Output Depth Image The Proposed Method Matching Input Depth Image High Resolution Database

Slide 32

The Proposed Method - Matching Matching Patches to database Matching is done between LR patches Kd tree is used for speeding up the process

Slide 33

The Proposed Method - Overview Construct Database Filter Input Generate Patches of input image Find Match Candidates Solve Minimum Energy Problem Reconstruct ImagePost Processing Filtering

Slide 34

The Proposed Method - Reconstruction Solving minimum energy problem Solved using TRW-S algorithm (based on belief propagation) E d -Unary Potential - Difference between normalized matching LR patches

Slide 35

The Proposed Method - Reconstruction E s -Pairwise Potential - Difference between un-normalized HR patch overlaps

Slide 36

The Proposed Method - Reconstruction yiyi yjyj E s - Pairwise Potential -

Slide 37

yiyi yjyj The Proposed Method - Reconstruction E s - Pairwise Potential -

Slide 38

yiyi yjyj The Proposed Method - Reconstruction E s - Pairwise Potential -

Slide 39

yiyi yjyj The Proposed Method - Reconstruction E s - Pairwise Potential -

Slide 40

-- E s =( + ( )2)2 )2)2 yiyi yjyj The Proposed Method - Reconstruction

Slide 41

Normalization is un-normalized based on the input patch min and max values

Slide 42

The Proposed Method - Overview Construct Database Filter Input Generate Patches of input image Find Match Candidates Solve Minimum Energy Problem Reconstruct ImagePost Processing Filtering

Slide 43

The Proposed Method Filter Results Noise reduction C. Post processing Denoising Outlier detection and correction using threshold Result Result after denoising input

Slide 44

Results - Qualitative Exp 1: Used Middleburry stereo dataset Down-sampled the ground truth (X2,X4) Reconstructed Compared RMSE Exp 2: 3 laser scans Upsampled by 4 Ground touth comparison Exp 3: Use synthetic Vs. real database

Slide 45

Results - Qualitative Proposed method Vs. Other Methods (Exp. 2) Proposed Method

Slide 46

Results - Qualitative Proposed method - Real Vs. Synthetic training data (Exp. 3)

Slide 47

Results - Quantitative Reminder: MRF RS Cross BilateralScSREbSR Upsampling factor Method used RMSE RGB-D image used Method used RGB-D image used

Slide 48

Results - Qualitative Results Movie

Slide 49

Conclusions 1 st or 2 nd best from intensity based methods MRF RS, Cross Bilateral are better but require more data Speed is not realtime compatible Super resolving moving depth videos Synthetic data exhibit better results than scanned data for training.

Slide 50

Future work Extend to exploit temporal context (in video) Exploit context when querying the database Develop a sensor specific noise model for better results

Slide 51

Paper review Pros Self contained Well referenced Novel approach Available resources website, data, code Good experimental results Cons Non chronological order of subjects (training after method, implementation notes after results, etc.) Supplied code insufficient

Slide 52

Questions?

Slide 53

References Some of the slides and images were taken from the original paper presentation, paper and website - http://visual.cs.ucl.ac.uk/pubs/depthSuperRes/ http://visual.cs.ucl.ac.uk/pubs/depthSuperRes/

Slide 54

Appendix - filters Bicubic filter - Interpolate data points on a two dimensional regular grid Smoother Fewer interpolation artifacts Uses 16 pixels (4X4) Bicubic Bilinear Nearest neighbor