4
Deblurring for Enlarged Image Shotaro Moriya, Satoshi Yamanaka, Yoshiki Ono, Koji Minami, and Hiroaki Sugiura Mitsubishi Electric Corporation Advanced Technology R&D Center Kyoto, Japan Abstract—When you use a high resolution display, input images are enlarged in some cases. Many chips designed for digital TVs and displays enlarge input images by linear methods and output blurred images. Therefore a deblurring method for an enlarged image is desired. We propose a new deblurring method for an enlarged image. Experimental results show the proposed method outputs a visually pleasant image. I. I NTRODUCTION When you use a high resolution display, input images are enlarged in some cases. Watching a DVD video on an LCD TV ready for HDTV is a typical example. If the input image is enlarged by a Linear enlargement method such as Bicubic [1] and Bilinear, an output image seems blurred because it cannot generate a higher frequency component than the Nyquist frequency of the input image. Sophisticated enlargement methods such as super resolution could solve this problem [2]. However many LSIs designed for consumer products implement linear enlargement methods and they output blurred images. Therefore signal processing that deblurs an enlarged image is desired. Greenspan suggested two kinds of signal processing based on the Laplacian pyramid [3], [4]. One is an enlargement method and the other is a deblurring method. His enlargement method is improved by some researchers [5]–[7]. This report focuses on his deblurring method and propose a new method that deblurs an enlarged image. II. CONVENTIONAL METHOD A. Greenspan’s Method The Laplacian pyramid is a sequence of high frequency component images obtained by a multiresolution analysis [8]. If there is an edge in an input image, there are zero crossing points at the same points in these high frequency component images and these images have a self-similarity around the zero crossing points [3], [4]. Greenspan thought if he found a nonlinear process that approximated to this self-similarity, a high frequency component that a blurred image didn’t have could be predicted by the nonlinear process. His nonlinear process is composed of two high pass filters (HPFs) and a clipping process. Let G 0 (x, y) denote a blurred image, The Greenspan’s deblurring method is described as follows. 1) Extract a high frequency component L 0 (x, y) by high pass filtering G 0 (x, y) by a first HPF. 2) Create L clip 0 (x, y) by clipping L 0 (x, y). This process is expressed as L clip 0 (x, y)= T if (L 0 (x, y) >T ) T if (L 0 (x, y) < T ) L 0 (x, y) otherwise (1) where T is a threshold. 3) Extract a high frequency component L 1 (x, y) by high pass filtering L clip 0 (x, y) by a second HPF. 4) Create deblurred image G 1 (x, y) by adding G 0 (x, y) and L 1 (x, y). B. Problems Greenspan’s deblurring method has some problems when it is applied to a linearly enlarged image. Fig. 1 depicts a block diagram of a linear upscaler. It is expressed as an upsampler followed by a low pass filter (LPF). Fig. 2a and 2b show Fourier spectrum of an input signal to the upscaler and an upsampled signal respectively. The Fourier spectrum of the upsampled signal is a periodic function because the upsampler replicates the Fourier spectrum of the input signal. In other words, the upsampler generates imaging components in the frequency domain [9]. The following LPF aims to cut these imaging components. If the LPF is ideal, the imaging components are removed completely as shown in Fig. 2c. But it is impossible to realize an ideal LPF and some of the imaging components remain as alias signals like Fig. 2d. If you apply the Greenspan’s deblurring method to an enlarged image, its first HPF extracts these alias signals. This is a first problem. Greenspan’s method uses the clipping process as a nonlinear process, and its recommended value for the threshold T in (1) is given by multiplying the maximum value of L 0 by a constant value [4]. However this threshold is not effective for many natural images because the clipping process works for an edge with the maximum contrast and some edges with close contrasts to it but doesn’t work for edges with small contrasts [5]. If the clipping process doesn’t work, his method is equal to a combination of two HPFs and extracts the imaging components only. Therefore it cannot make an effective high frequency component. This is a second problem. Additionally, even though the clipping process works, the second HPF that follows the clipping process might cause ringing [6]. This is a third problem. 2011 IEEE International Conference on Consumer Electronics - Berlin (ICCE-Berlin) 978-1-4577-0234-1/11/$26.00 ©2011 IEEE 249

[IEEE 2011 IEEE First International Conference on Consumer Electronics - Berlin (ICCE-Berlin) - Berlin, Germany (2011.09.6-2011.09.8)] 2011 IEEE International Conference on Consumer

  • Upload
    hiroaki

  • View
    216

  • Download
    2

Embed Size (px)

Citation preview

Page 1: [IEEE 2011 IEEE First International Conference on Consumer Electronics - Berlin (ICCE-Berlin) - Berlin, Germany (2011.09.6-2011.09.8)] 2011 IEEE International Conference on Consumer

Deblurring for Enlarged ImageShotaro Moriya, Satoshi Yamanaka, Yoshiki Ono, Koji Minami, and Hiroaki Sugiura

Mitsubishi Electric CorporationAdvanced Technology R&D Center

Kyoto, Japan

Abstract—When you use a high resolution display, input imagesare enlarged in some cases. Many chips designed for digital TVsand displays enlarge input images by linear methods and outputblurred images. Therefore a deblurring method for an enlargedimage is desired. We propose a new deblurring method for anenlarged image. Experimental results show the proposed methodoutputs a visually pleasant image.

I. INTRODUCTION

When you use a high resolution display, input images areenlarged in some cases. Watching a DVD video on an LCDTV ready for HDTV is a typical example. If the input imageis enlarged by a Linear enlargement method such as Bicubic[1] and Bilinear, an output image seems blurred becauseit cannot generate a higher frequency component than theNyquist frequency of the input image.

Sophisticated enlargement methods such as super resolutioncould solve this problem [2]. However many LSIs designed forconsumer products implement linear enlargement methods andthey output blurred images. Therefore signal processing thatdeblurs an enlarged image is desired.

Greenspan suggested two kinds of signal processing basedon the Laplacian pyramid [3], [4]. One is an enlargementmethod and the other is a deblurring method. His enlargementmethod is improved by some researchers [5]–[7]. This reportfocuses on his deblurring method and propose a new methodthat deblurs an enlarged image.

II. CONVENTIONAL METHOD

A. Greenspan’s Method

The Laplacian pyramid is a sequence of high frequencycomponent images obtained by a multiresolution analysis [8].If there is an edge in an input image, there are zero crossingpoints at the same points in these high frequency componentimages and these images have a self-similarity around thezero crossing points [3], [4]. Greenspan thought if he founda nonlinear process that approximated to this self-similarity,a high frequency component that a blurred image didn’t havecould be predicted by the nonlinear process. His nonlinearprocess is composed of two high pass filters (HPFs) and aclipping process. Let G0(x, y) denote a blurred image, TheGreenspan’s deblurring method is described as follows.

1) Extract a high frequency component L0(x, y) by highpass filtering G0(x, y) by a first HPF.

2) Create Lclip0 (x, y) by clipping L0(x, y). This process is

expressed as

Lclip0 (x, y) =

⎧⎪⎨⎪⎩

T if(L0(x, y) > T )

−T if(L0(x, y) < −T )

L0(x, y) otherwise

(1)

where T is a threshold.3) Extract a high frequency component L−1(x, y) by high

pass filtering Lclip0 (x, y) by a second HPF.

4) Create deblurred image G−1(x, y) by adding G0(x, y)and L−1(x, y).

B. Problems

Greenspan’s deblurring method has some problems whenit is applied to a linearly enlarged image. Fig. 1 depicts ablock diagram of a linear upscaler. It is expressed as anupsampler followed by a low pass filter (LPF). Fig. 2a and2b show Fourier spectrum of an input signal to the upscalerand an upsampled signal respectively. The Fourier spectrumof the upsampled signal is a periodic function because theupsampler replicates the Fourier spectrum of the input signal.In other words, the upsampler generates imaging componentsin the frequency domain [9]. The following LPF aims to cutthese imaging components. If the LPF is ideal, the imagingcomponents are removed completely as shown in Fig. 2c.But it is impossible to realize an ideal LPF and some of theimaging components remain as alias signals like Fig. 2d. Ifyou apply the Greenspan’s deblurring method to an enlargedimage, its first HPF extracts these alias signals. This is a firstproblem.

Greenspan’s method uses the clipping process as a nonlinearprocess, and its recommended value for the threshold T in(1) is given by multiplying the maximum value of L0 by aconstant value [4]. However this threshold is not effective formany natural images because the clipping process works foran edge with the maximum contrast and some edges withclose contrasts to it but doesn’t work for edges with smallcontrasts [5]. If the clipping process doesn’t work, his methodis equal to a combination of two HPFs and extracts the imagingcomponents only. Therefore it cannot make an effective highfrequency component. This is a second problem.

Additionally, even though the clipping process works, thesecond HPF that follows the clipping process might causeringing [6]. This is a third problem.

2011 IEEE International Conference on Consumer Electronics - Berlin (ICCE-Berlin)

978-1-4577-0234-1/11/$26.00 ©2011 IEEE 249

Page 2: [IEEE 2011 IEEE First International Conference on Consumer Electronics - Berlin (ICCE-Berlin) - Berlin, Germany (2011.09.6-2011.09.8)] 2011 IEEE International Conference on Consumer

Fig. 1. Diagram of an upscaler.

(a) Input signal. (b) Upsampled signal.

(c) Upsampled signal followed byideal LPF.

(d) Upsampled signal followed bynon-ideal LPF.

Fig. 2. Fourier spectrum. Fn means the Nyquist frequency of the inputsignal.

III. PROPOSED METHOD

In order to solve the first problem, we use a band pass filter(BPF) instead of the first HPF. The alias signals generated byan enlargement are cut by the BPF. Also we use another non-linear process reported by authors [7] instead of clipping. Thisprocess works for every zero crossing point, i.e. every edge.Therefore this process has stronger effect than Greenspan’smethod. The proposed method is described as follows.

1) Extract a horizontal frequency component HL0(x, y) byband pass filtering G0(x, y) by a horizontal BPF.

2) Create a nonlinearly processed signal HLnl0 (x, y) by

making slopes in the neighborhoods of zero crossingpoints in HL0(x, y) steeper. HLnl

0 (x, y) is given by

HLnl0 (x,y)=

⎧⎪⎨⎪⎩

k·HL0(x,y) if(HL0(x,y)·HL0(x+1,y)<0)

k·HL0(x,y) if(HL0(x,y)·HL0(x−1,y)<0)

HL0(x,y) otherwise(2)

where k is a constant.3) Create a horizontal high frequency component

HL−1(x, y) by high pass filtering HLnl0 (x, y).

4) Extract a vertical frequency component VL0(x, y) byband pass filtering G0(x, y) by a vertical BPF.

5) Create a nonlinearly processed signal VLnl0 (x, y) by

making slopes in the neighborhoods of zero crossing

(a) Part of original image.

(b) Part of GB0 , i.e. enlarged image by Bicubic.

(c) Part of GG−1, i.e. deblurred image by Greenspan’s method.

(d) Part of GP−1, i.e. deblurred image by the proposed method.

Fig. 3. Comparative results of the original image, GB0 , GG

−1, and GP−1.

These images are based on N6RGB(ISO-sRGB) in SCID [10].

points in VL0(x, y) steeper. VLnl0 (x, y) is given by

VLnl0 (x,y)=

⎧⎪⎨⎪⎩

k·VL0(x,y) if(VL0(x,y)·VL0(x+1,y)<0)

k·VL0(x,y) if(VL0(x,y)·VL0(x−1,y)<0)

VL0(x,y) otherwise(3)

6) Create a vertical high frequency component VL−1(x, y)by high pass filtering VLnl

0 (x, y).7) Create deblurred image G−1(x, y) by adding G0(x, y),

HL−1(x, y), and VL−1(x, y).

IV. EXPERIMENTAL RESULTS

We carried out an image simulation composed of followingsteps.

1) Reduce the width and the height of an original imageby half. We use the Bicubic method to resize image.

2) Make a blurred image GB0 by enlarging the reduced

image twice by Bicubic.3) Create a deblurred image GG

−1 by applying theGreenspan’s method to GB

0 .

250

Page 3: [IEEE 2011 IEEE First International Conference on Consumer Electronics - Berlin (ICCE-Berlin) - Berlin, Germany (2011.09.6-2011.09.8)] 2011 IEEE International Conference on Consumer

−2Fn −Fn 0 Fn 2Fn

−6

−4

−2

0

Frequency

Four

ier

Spec

trum

( lo

g sc

ale

)

BicubicGreenspanProposed

(a) Horizontal Fourier spectra.

−2Fn −Fn 0 Fn 2Fn

−6

−4

−2

0

Frequency

Four

ier

Spec

trum

( lo

g sc

ale

)

BicubicGreenspanProposed

(b) Vertical Fourier spectra.

Fig. 4. Comparative results of Fourier spectra. The dash line with circles (◦) is a Fourier spectrum of GB0 . The dash line with plus signs (+) is a Fourier

spectrum of GG−1. The dash line with dots (·) is a Fourier spectrum of GP

−1. Fn is the Nyquist frequency of the reduced image.

4) Create a deblurred image GP−1 by applying the proposed

method to GB0 .

5) Compare GB0 , GG

−1, and GP−1.

Fig. 3 shows comparative results of the original image,GB

0 , GG−1, and GP

−1. We use a luminance component ofN6RGB(ISO-sRGB) in SCID [10] as the original image. Colorconversion from RGB to luminance(Y) is expressed as [11]

Y = 0.299R+ 0.587G+ 0.114B (4)

GB0 seems blurred the most. GG−1 seems the blockiest. GP−1 is

visually pleasant the most.Fig. 4 show comparative results of horizontal Fourier spectra

and vertical Fourier spectra of GB0 , GG−1, and GP−1. A hori-

zontal Fourier Spectrum HH and a vertical Fourier SpectrumHV are calculated by

HH(fH) =∑fH

H(fH , fV ) (5)

HV (fV ) =∑fH

H(fH , fV ) (6)

where H(fH , fV ), fH , and fV mean a Fourier spectrum of adigital image, a horizontal frequency, and a vertical frequencyrespectively.

When you observe the Fourier spectrum of GB0 , you can

find local peaks in areas where fH or fV are larger thanFn or smaller than −Fn. These peaks are originally fromimaging components. These peaks becomes higher in theFourier spectrum of GG

−1. This means that Greenspan’s methodincreases the imaging component of GB

0 and is not suited foran enlarged image. On the other hand, these local peaks arevanished in the Fourier spectrum of GP

−1. This means thatthe proposed method generates a new frequency component

without increasing the imaging component. Thus the proposedmethod is preferable to Greenspan’s method.

V. CONCLUSION

We propose a new deblurring method that is suitable for anenlarged image. The experimental results show the proposedmethod outputs a visually pleasant image. Additionally it isconfirmed that the proposed method is able to generate aneffective high frequency component without increasing animaging component that an enlarged has in a Fourier spectrum.

REFERENCES

[1] R. G. Keys, “Cubic convolution interpolation for digital image process-ing,” IEEE Trans. Acoust., Speech, Signal Process., vol. 29, no. 6, pp.1153–1160, Dec. 1981.

[2] S. C. Park, M. K. Park, and M. G. Kang, “Super-resolution imagereconstruction : a technical overview,” IEEE Signal Process. Mag.,vol. 20, no. 3, pp. 21–36, May 2003.

[3] H. Greenspan and C. H. Anderson, “Image enhancement by non-linearextrapolation in frequency space,” in Proc. SPIE Image and Video II,Feb. 1994, pp. 2–13.

[4] H. Greenspan, C. H. Anderson, and S. Akber, “Image enhancementby nonlinear extrapolation in frequency space,” IEEE Trans. ImageProcess., vol. 9, no. 6, pp. 1035–1048, Jun. 2000.

[5] M. Shimizu, M. Nakashizuka, and Y. Iiguni, “Image enhancement andzooming by morphological operations in laplacian pyramid representa-tion,” IEICE Technical Report, vol. 106, no. 260, pp. 31–36, Sep. 2006.

[6] A. Shimura, K. Arakawa, and R. Taguchi, “A digital image enlargingmethod without edge effect by using epsilon-filter,” The Transactions ofthe Institute of Electronics, Information and Communication Engineers.A, vol. J86-A, no. 5, pp. 540–551, May 2003.

[7] S. Moriya, S. Yamanaka, S. Minami, and H. Sugiura, “Resolution en-hancement based on laplacian pyramid,” IEEE Trans. Consum. Electron.,vol. 56, no. 3, pp. 1830–1836, Aug. 2010.

[8] P. J. Burt and E. H. Adelson, “The laplacian pyramid as a compactimage code,” IEEE Trans. Commun., vol. 31, no. 4, pp. 532–540, Apr.1983.

[9] L. Tan, Digital Signal Processing : Fundamentals and Applications.Academic Press, 2008.

251

Page 4: [IEEE 2011 IEEE First International Conference on Consumer Electronics - Berlin (ICCE-Berlin) - Berlin, Germany (2011.09.6-2011.09.8)] 2011 IEEE International Conference on Consumer

[10] Graphic Technology – Prepress Digital Data Exchange – StandardColour Image Data (XYX/SCID), Japanese Standards Association JISX 9204:2004, 2004.

[11] Studio Encoding Parameters of Digital Television for Standard 4:3 andWide Screen 16:9 Aspect Ratios, ITU-R Recommendations BT.601-6,2007.

252