Fast Adaptive Graph-Cuts Based StereoMatching

Michel Sarkis, Nikolas Dorfler, and Klaus Diepold

Institute for Data Processing (LDV)Technische Universitat Munchen (TUM)

Munich, Germanymichel@tum.de, doerflen@in.tum.de, kldi@tum.de

Abstract. Stereo vision is one of the central research problems in com-puter vision. The most difficult and important issue in this area is thestereo matching process. One technique that performs this process isthe Graph-Cuts based algorithm and which provides accurate results [1].Nevertheless, this approach is too slow to use due to the redundant com-putations that it invokes. In this work, an Adaptive Graph-Cuts basedalgorithm is implemented. The key issue is to subdivide the image intoseveral regions using quadtrees and then define a global energy functionthat adapts itself for each of these subregions. Results show that the pro-posed algorithm is 3 times faster than the other Graph-Cuts algorithmwhile keeping the same quality of the results.

1 Introduction

Extracting depth information from stereo images is a very common researchtopic in Computer Vision. The main issue of stereo matching is to find densecorrespondences between the images from which the depth map of the scenecan be easily extracted. In the last decade, a number of different algorithms forstereo matching were developed. In [2], a very good review is presented alongwith a methodology to compare such algorithms.

The main issue in any stereo matching algorithm is to compute some costsusing similarity measures, and then define a suitable cost function upon whichthe minimum is the desired depth map. These algorithms are divided into threegroups of approaches depending on how the cost function is optimized.

Local-based optimization approaches like the adaptive windows techniquesare fast but prone to problems on occlusion boundaries [3,4]. Scanline-basedoptimization approaches like dynamic programming produce better results onocclusion boundaries and are also fast, but the results contain a lot of inconsis-tencies among the scanlines [2,5,6,7]. Global-based optimization approaches likethe Graph-Cuts avoid the disadvantages of the other groups and give an optimalsolution for all pixels at once [1,8]. However, these algorithms are very slow dueto the high complexity of the computations involved.

The Graph-Cuts based stereo matching approach in [1] minimizes a global en-ergy function. This formulation allows the disparity function to preserve discon-tinuities and to be piecewise smooth which leads to high reconstruction quality,

J. Blanc-Talon et al. (Eds.): ACIVS 2007, LNCS 4678, pp. 818–827, 2007.c© Springer-Verlag Berlin Heidelberg 2007

Fast Adaptive Graph-Cuts Based Stereo Matching 819

especially in the discontinuity regions. This algorithm generally requires mini-mizing a non-convex function with thousands of dimensions which is a NP-hardproblem. Hence, it requires a very high computational effort which leads to asignificant amount of processing time.

In this work, an adaptive Graph-Cuts based algorithm is presented. The keyissue is to subdivide the image into several regions using quadtrees [9], computethe costs adaptively for each subregion, then minimize a global energy functionwhich is adapted for each subregion. Results show that the proposed algorithmhave a faster convergence rate than the other Graph-Cuts algorithms due to thethe adaptivity of the cost function. This leads to an order 3 speed-up of thedepth map computation process.

Section 2 briefly presents the Graph-Cuts algorithm of [1]. Section 3 describesthe proposed Adaptive Graph-Cuts stereo matching algorithm. Section 4 showsan analysis and comparison of the proposed technique with that of [1]. Finally,conclusions are drawn in Section 5.

2 The Graph-Cuts Algorithm

Computing the depth map using Graph-Cuts is equivalent to finding the optimallabeling function f for every pixel p in the set of all pixels P . f labels every pixelp ∈ P with a discrete set of labels L. Each label l ∈ L corresponds to a certaindepth value. Therefore, a pair 〈p, l〉 corresponds to a single 3D-point in space.

The matching between two pixels in the left and right image is formulated interms of interactions between the pixels at the same depth label l. It requiresthat the corresponding 3D-points in an interaction must lie on the same depthlabel, i.e. if 〈p1, l1〉, 〈p2, l2〉 ∈ I then l1 = l2. An interaction is said to be activeif it is visible in both pixels p and q. The energy function used to compute theoptimal f is defined as:

E (f) = Edata (f) + Esmoothness (f) + Evis (f) . (1)

As can be noticed, the cost function is composed of three different parts. The pho-toconsistency term Edata forces the interacting points to have a similar intensityvalue. The smoothness term Esmoothness ensures that f is piecewise continuous;it implies that the neighboring pixels normally have the same disparity, exceptfor discontinuities at object borders. The visibility term Evis forces the visibilityconstraint to be taken into account [1].

The algorithm uses the α-expansion technique to get from a configuration fthe new configuration f ′. In f ′, a pixel get either relabeled with a new dispar-ity label or keep its old disparity label. An important issue is that the initialconfiguration has to conform with the disparity search constraints.

In order to find the α-expansion, a graph must be constructed. This graphhas two distinguished terminal nodes which are called the source and the sink.Each other node in this graph is connected to these two terminals and to itsneighbors by weighted edges. Now, finding the α-expansion move of a disparitylevel can be reduced to the problem of finding the minimal cut in this graph.

820 M. Sarkis, N. Dorfler, and K. Diepold

This is also equivalent to finding the maximal flow from the source to the sink.For this task, several algorithms can be used, i.e. [10,11]. This process is thenrepeated until each pixel is labeled with its optimal disparity label.

3 Adaptive Graph-Cuts Algorithm

3.1 The Adaptive Cost Function

The cost function defined in (1) is composed of three terms that can be usuallybe varied depending on the images that are used. The first term of (1), thephotoconsistency or data cost, is written as:

Edata = min((Vp (x, y) − Vq (x + l, y))2 − Kd, 0

), (2)

where Vp is the intensity value of the pixel p, Vq is the intensity value of thecandidate pixel q and l is the label or disparity value that is tested for thecandidate pixel q [1]. This equation presents the squared difference between theleft and right pixel intensities subtracted from some predefined regularizationconstant Kd that will be later determined automatically for each region in theimage. The minimum between the substraction result and 0 is taken to ensurethat the data cost is always non-negative. The smoothness term in its turn isdefined as:

Esmoothness = min (| lp − lp+1 |, Ks) , (3)

where lp is the label (disparity value) of the pixel p, lp+1 is that of its neighborpixel and Ks is the smoothness regularization constants.

The last term of (1) defines the visibility of a pixel. It will be set to zero ifthe pixel is visible; otherwise, it will be set to infinity.

In [1], the terms of the cost function are taken for the entire image. Theyare usually set manually depending on the image that is under study. In theproposed algorithm, however, the image will be subdivided into several regionsusing Quadtrees [9]. Thus, the regularization terms Kd and Ks will be varieddepending on each subregion.

The regions with low depth variation require more weight on the smoothnessconstraint since it is more probable that the pixels in these regions have similardisparity values and that they are not occluded. In the regions with high disparityvariation, the pixels have higher probability to be discontinuous and occluded.Therefore, it makes sense to vary the weights of the cost function depending onthe region.

Unlike [1], Kd and Ks cannot be set manually since each image contains severalregions. Therefore, the proposed algorithm will be conducted in a hierarchicalstructure. This means, that the stereo images will be downsampled into severallevels. At the coarsest level, the disparity map will be computed like proposedin [1]. Kd and Ks can be either manually set at the coarsest level or computedby using an algorithm that measures the variation of the pixels in the image.

Then, the disparity values found at this level will be used as a guide for thenext finest level. Depending on the disparity map found at the coarser level,

Fast Adaptive Graph-Cuts Based Stereo Matching 821

the stereo images will be subdivided using the Quadtrees. Then, for each regionor the leaf of the Quadtree, a statistical measuring algorithm will be employedto estimate the depth variation of the pixels and vary the parameters of thecost function accordingly to each of these subregions. One simple criteria thatmeasures the depth variation is the standard deviation of the disparities. Anothermeasure that can be used is the skewness of the pixels [12].

Once the parameters of the cost function are determined, the disparity searchin the next finest level will be conducted by refining the disparity values foundat the lowest level. This process will be repeated until the final level is reached.

3.2 Narrowband Disparity Refinement

Disparity values in the coarse disparity map are half the precision of those in thefiner map. So the upsampled disparity values at the next finer level are the onesobtained from the coarser level multiplied by 2. Then, the values in between willbe interpolated using the nearest neighbor interpolation technique [13].

To find the true value of the disparity at the next finer level, only a searchin a small search region, the narrowband, is necessary [14,15]. Therefore, theefficiency of the search will be highly increased by the a-priori knowledge of theguided optimization. To refine the disparity map, a narrowband matching volumelimited by dmin(x, y) and dmax(x, y) is initialized. Suppose that d′(x, y) is the dis-parity at position x,y in the upsampled disparity map. The true disparity df (x, y)can be found within a range drefine from d′(x, y) . Hence, the narrowband is lim-ited by dmax(x, y) = d′(x, y) + drefine and dmin(x, y) = d′(x, y) − drefine wheredrefine specifies the width of the interval around the estimated value from thecoarser level. The value of drefine depends on the disparity estimation error inthe coarser disparity map. In case if this error is below 1.5 pixels, then, drefine

in the finer disparity map should be multiplied by two, i.e. drefine = 3. Such asearch region can be visualized in the example shown in Fig. 1 a.

To only define a search region is not enough since at the coarse level a pixelrepresents a small neighborhood of pixels in the fine map. Consequently, if adiscontinuity occur at a position somewhere in this neighborhood, the searcharea should be expanded at this location to take that into account. In addition,occlusions might also occur at disparity discontinuities, and this has also to betaken into account. To overcome these problems, it is necessary to extend theinitial search region, e.g. the shown in Fig. 1 a, with an erode and dilate step.

– Erode: This step is applied to dmin(x, y). A new map d′min(x, y) is con-structed where every d′min(x, y) gets the minimum value of the neighbors ofdmin(x, y).

– Dilate: This step is applied to dmax(x, y). A new map d′max(x, y) is con-structed where every d′max(x, y) gets the maximum value of the neighbors ofdmax(x, y).

The final search region after the erode and delate steps of Fig. 1 a is shown inFig. 1 b. As noticed, the pixels at the edges have now a wider search range whichallows to overcome the problems mentioned before. The narrowband refinementalgorithm is illustrated in Table 1.

822 M. Sarkis, N. Dorfler, and K. Diepold

a) b)

Fig. 1. The narrowband disparity refinement region. a: Primary refinement region afterexpanding each disparity by dmax(x, y) = d′(x, y)+ drefine and dmin(x, y) = d′(x, y)−drefine. b: Final refinement region after the erode and dilate step.

Table 1. Disparity Refinement Algorithm

Step 1: Upsample the dc and scale the disparity values by 2.Step 2: calculate dmin = 2dc − drefine and dmax = 2dc + drefine.Step 3: Erode dmin and dilate dmax for every pixel.Step 4: Refine the disparity value for each pixel using the found region search.

3.3 Quadtree Subdivisions

The original Graph-Cut algorithm is designed to provide a global solution forthe whole image [1]. In order to apply the adaptive cost function defined in Sec-tion 3.1, the upsampled disparity map will be subdivided using Quadtrees so thatthe cost function can be separately adapted in each region. Another techniquethat can be equally used to divide the image is the rectangular subdivisions de-fined in [16] since it has also similar properties to the Quadtrees. Nevertheless,it was recently shown in [15] that the Quadtree subdivision is far more efficientthan the rectangular subdivision.

Given is a matching volume box B with dimensions W × H × D, where W isthe width of the disparity map, H is its height, and D is the number of disparitylabels. What has first to be done, is to split this box into 4 children boxes Bj

where j ∈ {1, 2, 3, 4}. Then, for each child Bj , the upper and lower disparitybounds Sj

min and Sjmax have to be computed as:

Sjmin = min (dmin (x, y))

Sjmax = max (dmax (x, y)) , (4)

over all x, y ∈ Bj . From these bounds, the costs for merging and splitting eachbox is then computed as

Fast Adaptive Graph-Cuts Based Stereo Matching 823

Cmerge (B) = w · h ·(max

(Sj

max

)− min

(Sj

min

))

Csplit (B) = w · h ·∑((

Sjmax

)−

(Sj

min

)), (5)

where w and h are the width and height of the box B. The first equation in (5),when minimized, reflects that the pixels have similar disparity labels and thuscan be merged together. The second one in (5) shows that the pixels have a lot ofvariation when minimized; hence, they have the potential to be split from eachother. Consequently, it is possible to compute the following cost function:

C (B) = min (Cmerge (B) , Csplit (B)), (6)

upon which it is decided whether a each box is either split into 4 new boxesor merged with other boxes. Notice that the quadtrees subdivisions try to findthe optimal divisions which minimizes the redundant calculations. Fig. 2 showsexamples of these subdivisions for the example given in Fig. 1 b. In Fig. 2 a, thewhole region is chosen. The search is done from S0

min to S0max. In Fig. 2 b, the

area is split in two subregions with ranges (S1min,S1

max) and (S2min, S2

max). InFig. 2 c the subdivisions found in Fig. 2 b are further subdivided while Fig. 2 dshows how some of these regions are merged back.

Figure 3 shows two generated tree structures for the Tsukuba data set [2]superimposed on the finest and coarsest levels. Notice that regions with higherdisparity differences were split into smaller regions than the ones with low dis-parity variation; a form which fits the narrowband refinement.

For each subdivision, the proposed Adaptive Graph-Cut algorithm is now ex-ecuted up to a predefined number of iterations. Since, the parameters of thecost function are adapted for each region, it might happen that the disparityfunction is not continuous among the neighboring regions. To resolve this defi-ciency, the parameters of the cost function are computed taken into account allthe neighboring divisions.

4 Results

The proposed Adaptive Graph-Cut algorithm will be compared with originalGraph-Cuts algorithm of [1]. In all the tests, the refinement parameter drefine

was set to 3 to account for a 1.5 pixel error on the coarser disparity map, thetree depth was set to 3, and a two level hierarchical refinement was conducted.

In these tests, two criteria will be checked: the speed of the algorithm andthe accuracy. The speed of the algorithms was measured using an AMD Duron2 GHz machine with 768 Mb RAM while the accuracy is tested using theMiddlebury stereo data set benchmark [17]. Both programs were written usingthe C language.

Table 2 shows the timing results of the proposed algorithm and that of [1]. Ascan be noticed, the proposed algorithm has outperformed the Graph-Cuts algo-rithm of [1]. To obtain the disparity maps of the Middlebury stereo benchmark,

824 M. Sarkis, N. Dorfler, and K. Diepold

a) b)

c) d)

Fig. 2. Dividing the narrowband signal of Fig. 1 b into several regions. a: The completeregion is chosen. b: The region is divided into two subregions. c: The regions found in bare further divided into more subregions. d: Some of the region found in c are merged.

Fig. 3. The generated tree structure for the Tsukuba data set. The left image is thetree generated on the finest level. The right image is the tree generated for the coarsestimage.

there is an average time improvement of 66%. This means that the proposed al-gorithm is three times faster which is a very significant amount of improvement.

In order to justify the improvement in time, it is also necessary to see the ac-curacy of the obtained results. In Table 3, the output of the Middlebury stereobenchmark is shown for both algorithms. In addition to that, the algorithm of [1]

Fast Adaptive Graph-Cuts Based Stereo Matching 825

is also presented while choosing the parameters of the cost function automatically.This was done to have a fair comparison since the parameters of the cost functionof the proposed Adaptive Graph-Cuts cannot be manually chosen.

When comparing the proposed algorithm to the automatic version of [1] wherethe parameters of the cost function were automatically chosen for the whole im-age, it can be noticed that both algorithms have almost the same performance.Thus combining this result with that of Table 3, it can be concluded that theproposed algorithm have the same accuracy performance as in [1] but can com-pute the disparity map of the scene three times faster. The obtained disparitymap on the Middlebury stereo set is visualized in Fig. 4.

Nevertheless, the results show that the ordinary Graph-Cuts algorithm withmanual choice of the cost function parameters have a better performance. Thisis due to the fact the proposed method to choose these parameters is not yetoptimal and need to be improved.

Table 2. Percentage improvement in the run-time between the proposed algorithmand that of [1]

Labels searched: 16 59 59 19Resolution: 384x288 450x375 450x375 434x383Data Set: Tsukuba Teddy Cones VenusProposed Algorithm 10.71 55.56 59.27 18.89Algorithm of [1] 33.55 155.29 187.49 55.98Improvement 68% 64% 68% 66%

Table 3. Evaluation results for the proposed algorithm and the algorithm of [1] usingMiddlebury stereo benchmark. GC designs the algorithm of [1], GC Auto designs thealgorithm of [1] while setting the parameters of the cost function automatically andAGC designs the proposed algorithm.

Algorithm Avg. Tsukuba Venus Teddy Conesground truth ground truth ground truth ground truth

Rank nonocc all disc nonocc all disc nonocc all disc nonocc all discGC 13.9 1.27 1.99 6.48 2.79 3.13 3.60 12.0 17.6 22.0 4.89 11.8 12.1

AGC 15.0 3.31 3.95 6.69 1.20 1.73 6.02 11.4 17.1 22.0 6.15 13.2 13.3GC Auto 15.1 2.33 3.03 9.07 1.12 1.56 5.50 10.4 15.8 20.4 8.20 14.6 15.7

5 Conclusion and Future Work

In this work, an adaptive Graph-Cuts algorithm was presented that determinesthe disparity map of a stereo image using an a adaptive cost function. Theadaptivity of the algorithm was obtained by splitting the image into severalregions using Quadtrees and then computing an adaptive cost function for eachof these regions. Results show that this scheme is three times faster than theother Graph-Cuts based stereo matching algorithms while keeping almost the

826 M. Sarkis, N. Dorfler, and K. Diepold

same accuracy. Looking into the future, a better function that measures thestatistics of the image should be implemented in order to further improve thequality results and maintain the current enhancement in speed.

Image Algorithm of [1] Proposed Algorithm

Fig. 4. Output of both algorithms using Tsukuba, Venus, Teddy and Cones image setsfrom the the Middlebury stereo benchmark

Acknowledgement. This research is sponsored by the German Research Foun-dation (DFG) as a part of the SFB 453 project, High-Fidelity Telepresence andTeleaction.

Fast Adaptive Graph-Cuts Based Stereo Matching 827

References

1. Kolmogorov, V., Zabih, R.: Multi-camera scene reconstruction via graph cuts. In:European Conference on Computer Vision (2002)

2. Scharstein, D., Szeliski, R.: A taxonomy and evaluation of dense two-frame stereocorrespondence algorithms. International Journal of Computer Vision 47(1) (2002)

3. Okutomi, M., Kanade, T.: A multiple-baseline stereo. IEEE Transactions on Pat-tern Analysis and Machine Intelligence 15(4) (1993)

4. Kanade, T., Okutomi, M.: A stereo matching algorithm with an adaptive window:Theory and experiment. IEEE Transactions on Pattern Analysis and Machine In-telligence, 16(9) (1994)

5. Crimsini, A., Shotton, J., Blake, A., Rother, C., Torr, P.: Efficient dense-stereo withocclusions and new view synthesis by four state DP for gaze correction. Technicalreport, Microsoft Research (2003)

6. Leung, C., Appleton, B., Sun, C.: Fast stereo matching by iterated dynamic pro-gramming and quadtree subregioning. In: British Machine Vision Conference (Sep-tember 2004)

7. Kim, J.C., Lee, K.M., Choi, B.T., Lee, S.U.: A dense stereo matching using two-passdynamic programming with generalized ground control points. In: IEEE Interna-tional Conference on Computer Vision and Pattern Recognition (CVPR), (2005)

8. Kolmogorov, V., Zabih, R.: Computing visual correspondence with occlusions usinggraph cuts. In: International Conference on Computer Vision (2001)

9. Balmelli, L., Kovacevic, J., Vetterli, M.: Quadtrees for embedded surface visualiza-tion: Constraints and efficient data structures. In: IEEE International Conferenceon Image Processing, (1999)

10. Roy, S., Cox, I.J.: A maximum-flow formulation of the n-camera stereo correspon-dence problem. In: International Confernce on Computer Vision (1998)

11. Boykov, Y., Veksler, O., Zabih, R.: Fast approximate energy minimization viagraph cuts. IEEE Transctions on Pattern Analysis and Machine Intelligence 23(11)(1999)

12. Kenney, J.F., Keeping, E.S.: Mathematics of Statistics. Van Nostrand, 3 rd edn.(1964)

13. Seul, M., O’Gorman, L., Sammon, M.J.: Practical Algorithms for Image Analysis:Descriptions, Examples, and Code, 1st edn. Cambridge University Press, Cam-bridge (2000)

14. Falkenhagen, L.: Hierarchical block-based disparity estimation considering neigh-bourhood constraints. In: International workshop on SNHC and 3D Imaging (Sep-tember 1997)

15. Leung, C.: Efficient Methods for 3D Reconstruction from Multiple Images. PhDthesis, University of Queensland (2005)

16. Sun, C.: Fast stereo matching using rectangular subregioning and 3d maximum-surface techniques. International Journal of Computer Vision 47(1), 99–117 (2002)

17. Scharstein, D., Szeliski, R.: www.middlebury.edu/stereo