A Fast Algorithm for Reconstruction-Based Superresolutionand Evaluation of Its Accuracy

Masayuki Tanaka and Masatoshi Okutomi

Graduate School of Science and Engineering, Tokyo Institute of Technology, Tokyo, 152-8550 Japan

SUMMARY

A superresolution process produces a high-resolutionimage from a set of low-resolution images. Reconstruction-based algorithms to produce the high-resolution imagewhich minimizes the difference between observed imagesand images estimated from the high-resolution image witha camera model have been developed. The reconstruction-based algorithm requires iterative calculation and has alarge calculation cost because reconstruction-based super-resolution is a large-scale problem. In this paper, a fastalgorithm for reconstruction-based superresolution is pro-posed. The proposed algorithm reduces the number ofobserved pixel value estimations from the high-resolutionimage, using an average of pixel values in a divided region.The effect of our proposed algorithm is demonstrated withsynthetic images and real images. The results show that theproposed algorithm is about 1.4 to 8.5 times faster thanconventional algorithms. © 2007 Wiley Periodicals, Inc.Syst Comp Jpn, 38(7): 44–52, 2007; Published online inWiley InterScience (www.interscience.wiley.com). DOI10.1002/scj.20662

Key words: superresolution; high speed; MAP;ML.

1. Introduction

A superresolution process reconstructs a high-reso-lution image (HRI) by combining multiple low-resolutionimages (LRIs). Numerous algorithms to accomplish thattask have been developed in the literature [1]. A maximum-

likelihood algorithm (ML) [2], a maximum a posteriori(MAP) algorithm [3], and projection onto convex sets(POCS) [4] are widely used. The ML algorithm is based onthe maximum-likelihood method. The MAP algorithm es-timates the HRI by maximizing the a posteriori HRI. POCSreconstructs the HRI by solving a linear system whichrepresents the image formulation from the HRI to LRIs. Analgorithm to reconstruct a color HRI from Bayer raw datausing MAP framework has also been proposed [5]. Theo-retical analyses of the fundamental limits are presented inRefs. 6 to 8.

These algorithms are classified as reconstruction-based algorithms which have a common framework. Thereconstruction-based algorithm estimates LRIs from theassumed HRI. Then, the HRI is updated to minimize thedifference between the observed LRIs and the estimatedLRIs. This process is iterated until convergence.

The reconstruction-based algorithm (1) is a verylarge-scale problem which has the same number of un-known parameters as the pixel number of the HRI, and (2)must estimate every pixel value of the LRIs. A closed formsolution is not feasible because of the very large scale of theproblem. An iterative method, such as the gradient descentmethod, is used for optimization because the optimizationproblem is a large-scale problem with numerous unknownparameters. The number of unknown parameters is typi-cally at least several thousand. The iterative method alsohas a high computational cost.

This paper proposes a novel and fast calculationalgorithm for MAP superresolution. We discuss only MAPbecause ML is a special case of MAP. However, the pro-

© 2007 Wiley Periodicals, Inc.

Systems and Computers in Japan, Vol. 38, No. 7, 2007Translated from Denshi Joho Tsushin Gakkai Ronbunshi, Vol. J88-D-II, No. 11, November 2005, pp. 2200–2209

44

posed algorithm can be applied to the ML and POCSmethods.

Nguyen and colleagues [10] used a preconditioningconjugate gradient method for optimization. This techniquereduces the number of iterations necessary in optimization.Elad and Or [11] proposed a fast superresolution algorithmassuming that the camera motion is pure translation. Theirfast superresolution algorithm consists of a measurementfusion process and deblurring process. The measurementfusion process generates a single blurred image from theLRIs. Then, the single blurred image is deblurred. Theiralgorithm has very limited application due to the restrictionof pure translational motion. For example, it cannot be usedfor a camera mounted on a vehicle or on a robot arm,because the camera motion is not solely translation. Incontrast, the HRI estimated by the proposed algorithm isidentical to the result of the conventional algorithm.

The proposed algorithm decreases the computationalcost by reducing the number of LRI pixel value estimations.It is remarkable that superresolution results obtained by theproposed algorithm are identical to results obtained by aconventional algorithm, even though the proposed algo-rithm requires less calculation than the conventional one.

The proposed algorithm discretizes and averages theLRI pixel values in a neighborhood. Because of this aver-aging, the cost function and its derivative are evaluated withless computation than the conventional algorithm. We ver-ify that discretization has a very small effect on the super-resolution result in comparison with the conventionalalgorithm.

Comparisons of the theoretical computational costand experimental calculation time show the effectivenessof our proposed algorithm. We also show theoretically andexperimentally that the results obtained by the proposedalgorithm are identical to those obtained by the conven-tional algorithm.

2. MAP-Based Superresolution

The MAP algorithm estimates the HRI by maximiz-ing the a posteriori probability. The cost function of thisoptimization problem consists of the sum of square differ-ences between the observed pixel values and the estimatedpixel values, and a constraint component. The conjugategradient method [14, 15] is usually used for optimization.In this section, we derive the conventional cost function ofthe MAP and ML algorithms.

2.1. Formulation

A certain pixel of the observation image is mappedto a certain point in the HRI space by registration. In otherwords, the registration transforms the observation imagesto LRI pixel data {xi, yi, fi}, where the suffix i is the pixel

number, (xi, yi) is the position of the pixel in HRI space, andfi is the pixel value. Figure 1 shows these models.

Convolution between the point spread function (PSF)and HRI produces the estimated pixel value of the observedLRIs. The cost function of the MAP algorithm consists oftwo components; the sum of square differences (SSD)between the estimated and observed pixel values and theconstraint component which represents the prior prob-ability. The cost function is expressed as

where h is a vectorized HRI, σ2 is the variance of theobservation noise, and b(xi, yi) is the vectorized PSF kernelfor position (xi, yi), Nl is the number of pixels of all LRIs,C is a matrix which represents the prior probability, and αis a constraint parameter. The number of pixels of all LRIsis equal to the product of the area of the region of interest(ROI) and the number of frames. The number of pixels ofall LRIs can be expressed as

The first component of Eq. (1) corresponds to alikelihood. The cost function of the ML algorithm is onlythe first component.

The gradient method is usually used for optimizationof the cost function. The derivative of the cost function inEq. (1) is

(1)

(2)

(3)

Fig. 1. Schematic diagram of superresolution process.

45

The computational cost of the constraint components inboth the cost function and its derivative is negligibly smallif we compare the SSD components. The evaluation of theSSD component requires Nl LRI pixel value estimations.This means that the computational cost of updating the HRIis proportional to the number of LRI pixel value estima-tions. In other words, we can accelerate the HRI reconstruc-tion by reducing the necessary number of LRI pixel valueestimations.

2.2. Superresolution using continuousregistration

The PSF vector b(xi, yi) depends on the position(xi, yi). All pixel positions of the LRIs are estimated withsubpixel precision. Thus, all PSF vectors are different. Wecall the superresolution used to reconstruct the HRI withcontinuous positions of the LRI pixels the algorithm usingcontinuous registration.

All LRIs are aligned to the LRI pixel data as shownin Fig. 2. The evaluation of the cost function for the algo-rithm using continuous registration requires pixel valueestimation of the black circles with continuous positions.

A large number of PSF vectors are needed to evaluatethe cost function. The number of PSF vectors is equal to thenumber of pixels of all LRIs. The same number of pixelintensity estimations are also required in order to evaluatethe cost function. In the reconstruction from 64 frames withan ROI size of 80 × 60, the number of pixels of all LRIs is307,200.

This algorithm can yield good results, since it doesnot use any approximations. However, this algorithm has avery high computational cost.

2.3. Superresolution using discrete registration

Discretization of the registered pixel positions can beused to reduce the high computational cost of the algorithmusing continuous registration. We call this superresolutionthe algorithm using discrete registration.

The algorithm using discrete registration estimatesthe pixel value at the white circle instead of the black circlein Fig. 2. The white circle represents the discretization grid.

When the ratio of the pixel size of the HRI to thediscretizing interval is an integer L, the number of necessaryPSFs is L2. For example, if L is 3, the number of necessaryPSFs is just 9. However, the evaluation of the cost functionstill requires Nl LRI pixel value estimations.

3. Fast Algorithm

The number of LRI pixel value estimations to evalu-ate the conventional cost function is equal to the totalnumber of LRI pixels. That number is typically in the tensor hundreds of thousands. We propose a new cost functionthat requires fewer LRI pixel value estimations than theconventional cost function.

The concept of the proposed algorithm is that theestimated LRI pixel value is compared with the average ofthe LRI pixel values in an HRI pixel area, whereas theconventional algorithm compares each LRI pixel value. Forexample, consider the three LRI pixel data in the shadedarea in Fig. 2. The conventional algorithm requires threeestimation iterations because there are three LRI pixel data.In contrast, the proposed algorithm requires only one esti-mation because there is one average regardless of the num-ber of LRI pixels. In this sense, the computational cost ofthe proposed algorithm for this shaded area is one-third thatof the conventional algorithm.

The variances of the average pixel values depend onthe number of pixels to be averaged, because the number ofregistration data in the neighborhood is different. Using theproperties of the variance, the error component of theproposed algorithm can be expressed as

where J1 represents the first component of the cost functionof the proposed algorithm, and

where j is the discretization grid number, Ng is the numberof the discretization grids, (xj, yj) is the position of thediscretization grid, Rj is the set of registration data in thej-th neighborhood area, Mj is the number of registration data

Fig. 2. Segmentation of observed nonuniformly spacedsamples. Black circles are continuous position of

registration data, white circle is discretized position ofregistration data, crosses represent discretization grid,

and shaded region is neighborhood region of thediscretization grid.

(4)

(5)

46

in the j-th neighborhood area, and σj2 is the variance of f

_j.

When the resolution enhancement ratio is Z, the number ofdiscretization grids Ng can be written as

where Wl is the width of the ROI and Hl is the height of theROI. The variance σj

2 of the mean can be expressed as [17]

The cost function can be written using Eqs. (4) and (7) as

The proposed algorithm optimizes cost function (8).The proposed algorithm must provide the same num-

ber of PSFs for superresolution using discrete registration.

3.1. Comparisons of computational cost(number of estimations)

The computational cost of the proposed algorithm isproportional to the necessary number of LRI pixel valueestimations.

Equation (8) shows that LRI pixel value estimationis needed only when Mj is greater than zero. Thus, thenecessary number of LRI pixel value estimations Ne is

where

If registration data exist in the neighborhood area,only one LRI pixel value estimation is required. This indi-cates that the number of LRI pixel value estimations of theproposed algorithm is smaller than or equal to the numberof discretization grids.

The case in which LRI pixel data exist in all neigh-borhood areas is the worst for the proposed algorithm. Inthat case, the number of estimations is equal to the numberof HRI pixels Nh. If all LRI pixel data exist alone in theneighborhood area, the number of estimations is the totalnumber of LRI pixels Nl. Hence,

Inequality (11) shows that the number of estimations for theproposed algorithm is less than or equal to that for theconventional algorithm in any situation.

The number of HRI pixels Nh is independent of thenumber of LRI frames. Consequently, the number of esti-mations of the proposed algorithm is never greater thanNh, even if the number of LRI frames increases. On the otherhand, the number of estimations in the conventional algo-rithm increases proportionally to the number of LRI frames,as shown in Eq. (2).

3.2. Equivalence to superresolution usingdiscrete registration

We now demonstrate that the superresolution resultsof the proposed algorithm are identical to the results of theconventional algorithm. The SSD component of the con-ventional evaluation function I1 can be expressed by usingthe SSD component of the proposed evaluation function J1

as

where

We provide a proof of Eq. (12) in the Appendix. Thedifference between I1 and J1 is merely a constant compo-nent whose value is determined before the superresolutionprocess. The gradients of I1 and J1 with respect to HRI hare identical because the cost function of the proposedalgorithm is identical to the conventional algorithm, exceptfor the constant component. Thus, both the proposed andconventional algorithms yield identical results.

The proposed algorithm yields identical results witha lower computational cost.

4. Experimental Results with SyntheticImages

We next compare the proposed algorithm with con-ventional algorithms using synthetic images. The calcula-tion time and precision are verified with differentdiscretization intervals and different numbers of LRIs.

4.1. Discretization interval versus calculationtime and precision

The calculation times for the proposed algorithm andthe algorithm using discrete registration were measuredwhile varying the discretization interval. The RMS errors

(7)

(8)

(10)

(11)

(9)

(12)

(6)

(13)

47

between the superresolution result and the ground truthwere also calculated.

The standard image, ISO/DIS 12640 (IOS 400), wasused as ground truth. We generated 64 LRIs from theground truth, where the PSF was assumed to be a Gaussianfunction with a standard deviation of 0.3 and the ROI sizewas 80 × 60. MAP superresolution is applied, with aresolution enhancement ratio of 3.2. Four neighboringMRFs were assumed for the constraint component [16].The corresponding matrix CT is a matrix which representsthe convolution operation with the Laplacian kernel shownin Fig. 3. The constraint parameter α is 0.3. The Fletcher-Reeves gradient method was used for optimization. Theinitial image for the optimization was the reference frameexpanded by bicubic interpolation. The calculation condi-tions are summarized in Table 1. A Pentium 4 computerwith a 2.8-GHz CPU was used for the calculation.

Figure 4 shows the ratio of the discretization intervalto the HRI pixel size versus the calculation time and theRMS error. The RMS error is common to the proposedalgorithm and the algorithm using discrete registration,because the two algorithms yield identical results.

The calculation time of the algorithm using discreteregistration is constant regardless of the ratio of the discreti-zation intervals. In contrast, the calculation time of theproposed algorithm increases until the discretization inter-val ratio is 5. When the discretization interval ratio is greaterthan 5, the calculation times of the two algorithms arealmost the same. The number of necessary LRI pixel valueestimations in the proposed algorithm increases with thediscretization grid interval ratio. Equation (11) shows thatthe number of necessary LRI pixel value estimations is theminimum of the total number of LRI pixels and the discreti-zation grids. When the discretization interval ratio is large,the number of necessary LRI pixel value estimations in theproposed method is close to the total number of LRI pixels.The total number of LRI pixels is equal to the number of

necessary LRI pixel intensity estimations in the algorithmusing discrete registration. Thus, the calculation times oftwo algorithms become closer when the discretization in-terval ratio is larger.

When the discretization interval ratio L is 1, thecalculation time is the minimum. In this case, the RMS erroris 3.08. For practical use, this RMS error is consideredsufficiently small.

Figure 5 shows the superresolution results when L =1, where (a) is the ground truth, (b) is one of the LRIs, (c)

Table 1. Calculation conditions

Fig. 3. Four-neighbor Laplacian.

Fig. 5. SR results from synthetic images.

Fig. 4. Relationship of discretizing grid interval tocalculation time and RMS error.

48

is the superresolution result of the algorithm using continu-ous registration, and (d) is the superresolution result of theproposed algorithm and the algorithm using discrete regis-tration. As discussed theoretically in Section 3.2, the pro-posed algorithm yields identical results to one of thealgorithms using discrete registration. The maximum dif-ference between the actual results of the two algorithms iszero. This means that the equivalence of the two algorithmsis validated experimentally. The calculation time, calcula-tion amount, and the RMS error versus the ground truth ofthree algorithms are shown in Table 2. We separate theprecalculation and optimization times from the calculationtime.

Figures 5(c) and (d) have better resolution than Fig.5(b) according to subjective assessment, especially thebottom right region of the image. The RMS error of theproposed algorithm is greater by 0.27 than that of thealgorithm using continuous registration. However, the re-sulting images are almost the same. Regarding calculationtime, the proposed algorithm is 12.2 times faster than thealgorithm using continuous registration and 6.2 times fasterthan that using discrete registration. The algorithm usingcontinuous registration needs a longer precomputation timeto prepare a huge number of PSFs. Compared with thealgorithm using discrete registration, the number of LRIpixel value estimations is reduced by 14.1% and the calcu-lation time is reduced by 8.2%. The reductions of thetheoretical computational cost and the actual calculationtime have the same trend.

4.2. Number of LRIs versus calculation timeand precision

The calculation time and the number of necessaryLRI pixel value estimations were measured while increas-ing the number of LRI frames, with a discretization intervalratio of 1. Figure 6 shows the measured calculation timeand the number of necessary LRI pixel value estimations.The RMS error is shown in Fig. 7; the RMS error is

common to the proposed algorithm and the algorithm usingdiscrete registration.

Figure 7 reveals that increasing the number of LRIframes decreases the RMS error. However, the calculationtime and computational cost of the algorithm using discreteregistration increase proportionally to the number of LRIframes. This means that increasing the number of LRIframes can decrease the RMS, but increases the calculationtime of the algorithm using discrete registration. In contrast,the calculation time and computational cost of the proposedalgorithm are roughly constant regardless of the number ofLRI frames. The number of necessary LRI pixel valueestimations in the proposed algorithm never exceeds thenumber of discretization grids. In fact, when the number offrames is greater than 100, the number of necessary LRIpixel value estimations is equal to the number of discreti-zation grids; in this case the number is 49,152. Even if thenumber of LRI frames increases, the computational cost ofthe proposed algorithm does not increase after 100 frames.This is shown in Eq. (11). The calculation times of thealgorithm using discrete registration with 100 frames and

Table 2. Comparison of three calculation methods forsynthetic images

Fig. 6. Relationship between number of LRIs andcalculation time by proposed method and discrete

registration method.

Fig. 7. Relationship between number of LRIs and RMSby proposed method.

49

with 200 frames are 19.5 and 39.5 s, respectively. In con-trast, the calculation times of the proposed algorithm are2.3 and 2.4 s, respectively. Although the speed-up dependson the number of frames, the proposed algorithm is 8.5times faster for 100 frames and 16.5 times faster for 200frames.

Figure 6 shows the strong correlation between thenumber of necessary LRI pixel value estimations and thecalculation time. The correlation factors of the algorithmusing discrete registration and the proposed algorithm are1.00 and 0.99, respectively. This figure verifies experimen-tally that the calculation time is proportional to the numberof necessary LRI pixel value estimations.

5. Experimental Results with Real Images

We applied the proposed algorithm to a real imagesequence captured by a hand-held camera. Dragonfly(Point Grey) was used for capture. The HRI was recon-structed from 32 frames and the discretization interval ratiowas 1. The other conditions were the same as in Table 1.*

Registration was performed assuming planar perspectivetransformation by the gradient descent method [13].

Figures 8, 9, and 10 show the expanded referenceframe of the LRI sequence, the superresolution result givenby the algorithm using continuous registration, and theresult of the proposed algorithm, respectively. Each figureincludes the whole image and the zoomed image. The resultof the algorithm using discrete registration is omitted be-cause the result is identical to that of the proposed algo-rithm. The warping effect of PSF is considered in thealgorithm using continuous registration.

The calculation time comparison is summarized inTable 3. In the experiment using real images, the proposedalgorithm is 2.1 times faster than the algorithm using dis-crete registration and 6.7 times faster than the algorithmusing continuous registration.

The RMS error of the proposed algorithm relative tothe algorithm using continuous registration consideringPSF warp is 1.07. This value means that the two results arealmost identical.

The experiments demonstrate the effectiveness of theproposed algorithm.

6. Conclusions

In this paper, we have proposed a fast MAP superre-solution algorithm which reduces the computational cost ofupdating the HRI.

*The constraint parameter α and the matrix Ct should be optimized foreach image. However, parameter optimization itself is a challengingproblem. In addition, the main topic of this paper is a fast algorithm. Forthese reasons, we use the parameter values that were chosen experimen-tally.

Fig. 8. Sample of observed image.

Fig. 9. SR result with continuous registration for realimage.

Fig. 10. SR result with discrete registration andproposed method for real image.

Table 3. Comparison of three calculation methods forreal images

50

The proposed algorithm defines the discretizationgrids and the neighborhood area. The average pixel valueof the pixels in the neighborhood area is used for thecalculation. The SSD component of the cost function of theproposed algorithm is the square difference between theaverage pixel value and the pixel value estimated for thediscretization grids from the assumed HRI. By using theproperties of the variance, the proposed algorithm canachieve results identical to the conventional algorithm.

It was experimentally validated that the computa-tional cost of superresolution is proportional to the numberof LRI pixel value estimations needed to evaluate the costfunction.

Experiments using real images captured with a hand-held camera are reported. They confirm that the proposedalgorithm makes the superresolution process much faster.Moreover, we verified quantitatively that the superresolu-tion images yielded by the proposed algorithm and theclassical algorithm were almost identical.

REFERENCES

1. Sung CP, Min KP, Kang MG. Super-resolution imagereconstruction: A technical overview. IEEE SignalProcess Mag 2003;26:21–36.

2. Tom BC, Katsaggelos AK. Reconstruction of a high-resolution image by simultaneous registration, resto-ration, and interpolation of low-resolution images.Proc IEEE Int Conf Image Processing 1995;2:539–542.

3. Schulz RR, Stevenson RL. Extraction of high-reso-lution frames from video sequences. IEEE TransImage Process 1996;5:996–1011.

4. Stark H, Oskoui P. High resolution image recoveryfrom image-plane arrays, using convex projections. JOpt Soc Am A 1989;6:1715–1726.

5. Goto T, Okutomi M. Direct super-resolution andregistration using raw CFA images. Proc IEEE Com-puter Soc Conf CVPR 2004;II:600–607.

6. Baker S, Kanade T. Limits on super-resolution andhow to break them. IEEE Trans Pattern Anal MachIntell 2002;24:1167–1183.

7. Lin Z, Shum HY. Fundamental limits of reconstruc-tion-based superresolution algorithms under localtranslation. IEEE Trans Pattern Anal Mach Intell2004;26:83–97.

8. Tanaka M, Okutomi M. Theoretical analysis aboutlimitations on reconstruction-based super-resolution.IPSJ SIG Technical Report, 2004-CVIM-147C, p147–154, 2005. (in Japanese)

9. Tanaka M, Okutomi M. Fast algorithm for recon-struction-based super-resolution. IPSJ SIG TechnicalReport, 2004-CVIM-146C, p 97–104, 2004. (in Japa-nese)

10. Nguyen N, Milanfar P, Golub G. A computationallyefficient superresolution image reconstruction algo-rithm. IEEE Trans Image Process 2001;10:573–583.

11. Elad M, Or YH. A fast super-resolution reconstruc-tion algorithm for pure translational motion and com-mon space-invariant blur. IEEE Trans Image Process2001;10:573–583.

12. Zitová B, Flusser J. Image registration methods: Asurvey. Image Vis Comput 2003;21:977–1000.

13. Baker S, Matthews I. Lucas-Kanade 20 years on: Aunifying framework. Int J Comput Vis 2004;56:221–255.

14. Hardi RC, Barnard KJ, Armstrong EE. Joint MAPregistration and high-resolution image estimation us-ing a sequence of undersampled images. IEEE TransImage Process 1997;6:1621–1633.

15. Press WH, Teukolsky SA, Vetterling WT, FlanneryBP. Numerical recipes in C++. Cambridge UniversityPress; 2002.

16. Capel D. Image mosaicing and super-resolution.Springer-Verlag; 2004.

17. Miyakawa M. Statistics technique. Kyoritu; 1998. (inJapanese)

APPENDIX

Proof of Eq. (12)

The error component of the conventional evaluationfunction I1 can be rewritten as

Using Eq. (14), we can simplify the difference between I1

and J1 to

By rewriting Eq. (15), we obtain Eq. (12).

(14)

(15)

51

AUTHORS (from left to right)

Masayuki Tanaka (member) received his bachelor’s and master’s degrees in control engineering and Ph.D. degree fromTokyo Institute of Technology in 1998, 2000, and 2003 and joined Agilent Technology. He has been a research scientist at TokyoInstitute of Technology since 2004.

Masatoshi Okutomi (member) received his B.E. degree in mathematical engineering and information physics from theUniversity of Tokyo in 1981 and M.E. degree in control engineering from Tokyo Institute of Technology (TITech) in 1983 andjoined the Canon Research Center. From 1987 to 1990, he was a visiting research scientist with the School of Computer Scienceat Carnegie Mellon University, Pittsburgh, USA. Based on his research into stereo vision at CMU, he received a Ph.D. degreefrom TITech in 1993. From 1994 to 2002, he was an associate professor at the Graduate School of Information Science andEngineering, TITech, and has been a professor at the Graduate School of Science and Engineering since 2002. His currentresearch interests include both theoretical aspects of computer vision and its novel applications. He is a member of theInformation Processing Society of Japan, the Robotics Society of Japan, the Institute of Image Electronics Engineers of Japan,IEICE, and IEEE.

52