1

Window-based Approach For Fast Stereo Correspondence

Raj Kumar Gupta, Siu-Yeung Cho

IET Computer Vision,  2013

2

Outline•Introduction•Related Work•Proposed Method•Experimental Results•Conclusion

3

Introduction•Using two correlation windows to improve the performance of the algorithm •3*3 and 9*9

•Real-time suitability •more than 10 frame/s on CPU in case of 320 × 240-sized image pair with disparity value 16

4

Related Work•Local methods are usually base on correlation.•Area-based (NCC, SAD, SSD)•Feature-based: rely on feature extraction and match local cues (BF, GF)

•Bigger window size, more information, more blurred.

5

Outline•Introduction•Related Work•Proposed Method•Experimental Results•Conclusion

6

Flow Chart

7

Flow Chart

8

Initial Matching•Matching cost computation: SAD

Left Right

d

9

Problem in disparity selection

a. Determine disparity easily for unique minimum valueb. Ambiguous disparity in case of multiple minimac. Matching cost calculated at point (205, 230) of Tsukuba image

10

Initial Matching: large correlation window

•Matching cost computation: SAD + penalty

•Penalty term

• Disparity computation

11

Problem in disparity selection

12

Initial Matching: small correlation window

•Only those disparity values that are carried by neighbouring pixels.•Matching cost computation without penalty

•N: the disparity values of the neighbouring pixels.•Avoid local minima and speed up

13

Flow Chart

14

Unreliable pixel detection

• left–right cross-checking

15

Disparity Interpolation•Search for pixels with reliable disparity value in its eight neighbouring pixels.•Compute similarity of unreliable pixel and its reliable neighbor.

16

Flow Chart

17

Disparity Refinement

18

Outline•Introduction•Related Work•Proposed Method•Experimental Results•Conclusion

19

Experimental Results•Computation time of the proposed algorithm for different window sizes on Tsukuba image.(image size 384 × 288 with 16 disparity labels)

20

• Percentage error in non-occluded (nocc), whole image (all) and near depth discontinuities (disc) for different window sizes for all four images (Tsukuba, Venus, Teddy and Cones)

21

Experimental Results

• a. Without using small correlation window Ws and the disparity refinement step• b. Without using the disparity refinement step• c. Without using small correlation window Ws• d. With all four steps on Tsukuba image

22

Experimental Results

23

24

25

Experimental Results•Comparison the performance of the proposed algorithm with other correlation-based algorithms.

26

Experimental Results

27

Reference• [24] Gupta, R., Cho, S.-Y.: ‘Real-time stereo matching using adaptive binary window’

(3D Data Processing, Visualization and Transmission, 2010)• [25] Zhang, K., Lu, J., Lafruit, G., Lauwereins, R., Gool, L.V.: ‘Real-time accurate

stereo with bitwise fast voting on Cuda’. Int. Conf. Computer Vision Workshops, 2009, pp. 540–547• [26] Humenberger, M., Zinner, C., Weber, M., Kubinger, W., Vincze, M.:‘A fast

stereo matching algorithm suitable for embedded real-time systems’, Comput. Vis. Image Underst., 2010, 114, (11),pp. 1180–1202• [27] Gong, M., Yang, Y.: ‘Near real-time reliable stereo matching using

programmable graphics hardware’. IEEE Conf. Computer Vision and Pattern Recognition, 2005, pp. 924–931• [28] Richardt, C., Orr, D., Davies, I., Criminisi, A., Dodgson, N.: ‘Real-time

spatiotemporal stereo matching using the dual-cross-bilateral grid’. European Conf. Computer Vision, 2010, vol. 6313, pp. 510–523• [29] Ambrosch, K., Kubinger, W.: ‘Accurate hardware-based stereo vision’,Comput.

Vis. Image Underst., 2010, 114, (11), pp. 1303–1316

28

29

30

31

32

33

34

Conclusion•A new correlation-based stereo-matching approach.• Large window improves at non-textured image regions• Small window improves at depth discontinuities

•The CPU implementation computes at a speed of more than 10 frame/s.•Easily implemented on GPU. •The proposed method can be used in real-time applications to reconstruct the 3D structures with great accuracy at object boundaries.

35

Codebook based Stereo Matching for Natural User

Interface

Sung-il Kang and Hyunki Hong

2013 IEEE International Conference on Consumer Electronics (ICCE)

36

Outline•Introduction•Proposed Method•Experimental Results•Conclusion

37

Introduction

•Interactive user interface has been one of the major topics in consumer electronics.•Gesture based user interface• Interactive smart TV, Nintendo Wii, Sony PlayStation3 Move, and Microsoft Kinect.

•Propose a stereo system implemented on GPGPU for real-time performance.•Employ codebook to solve occlusion.

38

Flow chart

39

Proposed Method•Pre-processing• Laplace od Gaussian (LoG) filter for alleviating the lighting effects.

•Cost initialization• AD+Census[6]

[6] X. Mei, X. Sun, M. Zhou, H. Wang, and X. Zhang, “On building an accurate stereo matchng system on graphics hardware,” Proc. of GPUCV, pp. 467-474, 2011. http://www.camdemy.com/media/4724

40

Proposed Method•Cost aggregation[6]

• Cross-based aggregation• Color similarity and the length constraint

•Refinement[6,7]

• Left-right consistency check• Iterative region voting• Sub-pixel enhancement

d

[7] Q. Yang, C. Engels, R. Yang, H. Stewenius, and D. Nister, “Stereo matching with color-weighted correlation, hierarchical belief propagation and occlusion handling,” IEEE Transactions on PAMI, 2009.

41

Proposed Method

Occlusion? Find codeword

Updatecodeword

Codeword?

Yes

No

Yes

No

Add a newcodeword

42

Experimental Results• Device: Intel Quad 2.66GHz with Nvidia GTX460. • Stereo images are captured by a Bumblebee 3 from Point Grey Inc. • Time: 80~110ms/frame• Stereo matching is implemented on GPU.• The codebook generation and its evaluation is on CPU.

43

Experimental Results

44

Experimental Results

[8] K. J. Yoon and I. S. Kweon, “Adaptive support-weight approach for correspondence search,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 28, pp. 650-656, 2005.[9] C. Richardt, D Orr, I Davies, and A Criminisi, “Real-time spatiotemporal stereo matching using the dual-cross-bilateral grid,” Proc. of ECCV, 2010.

45

Conclusion•Propose a stereo system implemented on GPGPU for real-time performance.•Good performance at static background Only.