Digital restoration of incoherent bandlimited images

Digital restoration of incoherent bandlimited images

Junji Maeda and Kazumi Murata

This paper describes the digital restoration method of incoherent bandlimited images in the presence ofnoise. The restoration procedure consists of spatial frequency filtering and a superresolution algorithm.The superresolution or the spectral extrapolation is performed by using both the Gerchberg algorithm andour modified one. The modified technique employs a priori information based on prototype images andprovides more desirable results than the original Gerchberg algorithm. Some results of computer simula-tions are presented which indicate the effectiveness and several problems of the proposed method.

1. Introduction

The problem of restoring the details of bandlimitedimages or superresolution has long been investigated bydigital methods.1-7 Among the several methods, aniterative restoration technique using a spectral ex-trapolation algorithm was developed by Gerchberg, 8 inwhich he applied his algorithm to 1-D coherent imagery(two-point resolution). Papoulis9 proposed the samealgorithm emphasizing signal extrapolation rather thanspectral extrapolation. Sabri and Steenaart' 0 pre-sented a matrix method for signal extrapolation. Theiterative signal extrapolation algorithm was generalizedto two dimensions and implemented on a coherentprocessor by Marks.11 12 Smith and Marks13 haveshown some examples of implementation of the closedform image extrapolation. Rushforth and Frost141 5

applied the Gerchberg algorithm and their new tech-nique to speckle interferometry. Youla16 has discusseda more general alternating orthogonal projection algo-rithm which contains the Gerchberg algorithm. Starket al. 17 described the restoration of finite-energy opticalobjects from two projections. Cahana and Stark1 8 haveshown new approaches to improve the rate of conver-gence of the Gerchberg algorithm. Recently, Maitre' 9

has shown the similarity between the Gerchberg algo-rithm and the Jacobi method with constraints.

In this paper we investigate the use of the Gerchbergalgorithm and our modified one for the restoration ofincoherent bandlimited images in the presence of noise.In other words we present 2-D digital superresolutionin the case of incoherent imaging through a diffrac-tion-limited pupil. Our modification is aimed at ob-taining the restored estimates with higher resolutionthan using the original Gerchberg algorithm. We havemodified the algorithm by incorporating additional apriori information based on prototype images and haveobtained desirable results. After discussing the res-toration procedure, we present the results of computersimulations.

11. Restoration Procedure

Consider an incoherent imaging process through aspace-invariant optical system with a diffraction-lim-ited clear pupil. The bandlimited image g (x ,y) is rep-resented by the convolution of an ideal image f(x,y)with a point spread function h (x,y) of the imagingsystem

g(x,y) = f(x,y) * h(xy) + n(x,y), (1)

where n(xy) is additive noise. In the spatial frequencyplane Eq. (1) becomes

G(u,v) = F(u,v) H(u,v) + N(u,v), (2)

where G, F, H, and N are the Fourier transforms of theg, f, h, and n, respectively. Assuming a square apertureof width d, the optical transfer function H(u,v) of theincoherent imaging system is20

[(i ...i"i) (1 -Ž L1 Jul < w'lvI < w,H(uv) = w o w (3)

0O, otherwise,The authors are with Hokkaido University, Faculty of Engineering,

Department of Applied Physics, Sapporo, Japan.Received 2 October 1981.0003-6935/82/122199-06$01.00/0.© 1982 Optical Society of America.

where w is a cutoff frequency of the system and is givenby w = d/Xz for wavelength X and image distance z.

The restoration method consists of spatial frequency

15 June 1982 / Vol. 21, No. 12 / APPLIED OPTICS 2199

(3)

filtering and a superresolution algorithm. The cor-rected image Ax,y) by spatial frequency filtering canbe written by

/(x,y) = 1-'jP(u,v)j = 5-YjG(u,v)- Y(u,v)l, (4)

where -1{ denotes inverse Fourier transformation,and Y(u,v) represents some filter function such as theinverse filter, the Wiener filter, etc. This filtering isintended to approach the filtered frequency responseto 1.0 at all frequencies within the limited band. Al-though the filtered image Ax,y) is a final output forconventional restoration, it is an initial input for theiterative superresolution algorithm. The procedure torestore bandlimited images is shown in Fig. 1. Theupper half of this figure represents conventional fil-tering of Eq. (4). The lower half (inside the dashed line)shows the iterative superresolution algorithm ofGerchberg, which is described below.

At first it is assumed that the extent of the ideal image(Lx and LY in this case) is known a priori. We definethe spatial truncation operator (spatial gate) D in theimage plane by'5

D,(x,y) = J(xy) Ixl L/2,lyI < Ly/2,l0, otherwise,

(5)

and the bandlimiting operator (perfect low-pass filter)B in the spectral plane by

fBP(u,v) = A(u'v), ul w'Jv < , (6)l, otherwise.

The Fourier transform pair of these operators, namely,the space truncation operator D in the spectral planeand the bandlimiting operator B in the image plane, aredefined by

Y-jDP(uV)j = Ax~y), Ix • LX/2, lyI < Ly/2, (7)l0, otherwise,

5{B,(xy) = (~uv) ul S w, v I w, (8)lo, otherwise,

respectively. In other words, operators D and B per-form a convolution with the sinc function in eachplane.

Using these operators and denoting the filtered imageAx,y) as go(x,y) and the filtered spectrum P(u,v) asGo(u,v), the iterative algorithm can be described asfollows:

Step 1. Truncate the filtered image by multiplyingby D (image constraints).

Step 2. Fourier transform.Step 3. Set the central portion of the spectrum to

zero by multiplying by (I - B) and then this portion isreplaced by the filtered spectrum Go(u,v), where I de-notes the identity operator in the spectral plane (spec-tral constraints).

Step 4. Inverse Fourier transform.Step 5. Implement the truncation by D, go to Step

2, and repeat.The restored image after the kth iteration can be

written from Step 515:

gk+i(x,y) = Djgo(xy) + (I-B)Dgk(x,y)j fork = 1,2,..., (9)

with the initial estimate

-I

I'k.

c r

L…-- - - - - - - - - - - - - - - - - - - - - - - - J

Fig. 1. Schematic block diagram of the restoration procedure ofincoherent bandlimited images.

g1 (x,y) = Dgo(x,y), (10)

where I denotes the identity operator in the imageplane. Similarly, the extrapolated spectrum after thekth iteration is given from Step 3 by8

Gk(U,v) = Go(u,v) + (I- B)DGk-l(u,v) fork = 1,2,.... (11)

It has been proved that an infinite number of itera-tions leads to perfect extrapolation in the case of co-herent noise-free imaging.8 9 16'2' In practice, however,the effects of quantization error, round-off error, andtruncation of iteration may cause imperfect extrapo-lation even in the absence of noise.9 The presence ofnoise will further reduce the degree of restoration since(1) the filtered spectrum Go(u,v) will inevitably containsome error due to noise and so will be the initial estimateg1(x,y) and (2) the error will be amplified with iterationbecause of the ill-conditioned nature of the algorithm.' 5

We have modified the Gerchberg algorithm to obtainimproved restored images with higher resolution in thepresence of noise.

Our modification is based on the concept of prototypeimages22-24 which are noise-free blur-free images withcontent similar to the ideal image. We have incorpo-rated new a priori information into Step 3 of the algo-rithm. More precisely, the magnitude of the secondterm on the right-hand side of Eq. (11) is replaced by theaverage of the spectral magnitudes of several prototypeimages, and the phase of this term is left unchanged.We have tried to increase the information of the Fourier

2200 APPLIED OPTICS / Vol. 21, No. 12 / 15 June 1982

transform magnitude outside the limited band. As forthe phase, it is indicated that the Fourier transformphase contains many of the features of its counterpart.25

Indeed, our early attempts to incorporate both thespectral magnitude and the spectral phase of prototypeimages yielded worse restoration than using only thespectral magnitude, since the features of prototypeimages appeared in the restored estimate. Thus wehave preserved the phase of the second term of Eq.(11).

The main problem of our modified method is how toselect appropriate prototype images. Although thedefinite standards to determine the best prototype havenot been found, the necessary conditions that a proto-type image should satisfy are (1) it belongs to the sameclass of ensemble as an ideal image and (2) it containssufficient spatial frequency components beyond thecutoff frequency. As to the first condition, we cannotknow exactly the ideal image itself. However, from thefiltered image Mx ,y) we can in general estimate its class.It has been demonstrated in our simulation that a va-riety of images of the same class which satisfy condition(2) provides almost the same restoration, since the av-erage of the spectral magnitudes of prototypes is usedin the algorithm.

Ill. Results of Computer Simulations and Discussion

We have applied both the Gerchberg algorithm andour modified one to the restoration of incoherent band-limited images and have made some comparisons be-tween them. The restored images are evaluated by twocriteria: one is the generated spectral energy (GSE)which is the produced energy outside the limited band,8

and the other is the restoring coefficient (R) which isdefined as26

R =11 SfI(X,y) - 9k+l(xy)1dxdy x 100(%) (12)f ff(x,y) - g(xy)) 2dxdy

for after the kth iteration. This R indicates how therestored image approaches the ideal one in comparisonwith the filtered one. Although all the functions pre-sented are continuous for convenience, the actualcomputations use discrete arrays with 64 X 64 samplingpoints in the image or the spectral plane. The com-puting time for one iteration cycle in the algorithm is-0.4 sec on the HITAC M-200H computer.

Figure 2 shows the model object consisting of athree-bar chart used as the ideal image f(xy) in oursimulation, in which Lx and Ly equal 33 samplingpoints. We limited the band of the ideal image by arectangular pupil having the OTF of Eq. (3) with 9points cutoff frequency, and then added zero-meanwhite Gaussian noise. The resulting incoherentbandlimited image g(x,y) at a SNR of 30 dB is shownin Fig. 3(a), in which the details of the resolution chartare totally lost. The SNR is defined by

S/N = 10. log variance of g(x,y) (13)

variance of n(xy)

The filtered output Ax ,y) obtained by using the inversefilter because of the relatively high SNR is shown in Fig.3(b). The low spatial frequency regions (upper left and

= II I- - u

Fig. 2. Model object used in the computer simulation.

(a) (b)

Fig. 3. (a) Incoherent bandlimited image at a SNR of 30 dB. (b)Filtered image by inverse filtering.

0

0. U0

Fig. 4. Log power spectra of the model object (- - -), the incoherentbandlimited image (---), and the filtered image (-).

lower right) are restored to a great extent, but the highfrequency parts (upper right and lower left) are still notresolved. Figure 4 represents logI F12, logI G 12, andlog 2, namely, the log power spectra of the idealimage, the bandlimited one, and the filtered one, re-spectively. This figure demonstrates that the filteredspectrum is nearly completely corrected within thelimited band.


" - _. =1111111 a

(a)

GSE

1.0

- -

_ .1-11 (b

(b) 0.Fig. 5. Restored images obtained by using (a) the original Gerchberg 1 10 20 30 40 50algorithm and (b) the modified algorithm with the single replacement Fig. 7. GSE curves of the original Gerchberg algorithm ( --) and

of a priori information. the modified algorithm with the single replacement of a priori in-formation (-).

1.0

Ill Ill

Fig. 6. One of the prototype images used in the modifiedalgorithm.

We then implemented the iterative extrapolationalgorithm by using the filtered image in Fig. 3(b) as theinitial input. The restored images obtained using theoriginal Gerchberg algorithm and our modified one afterfifty iteration cycles are shown in Figs. 5(a) and (b),respectively. Although the Gerchberg algorithm givesconsiderable restoration from the filtered image (R =55.6%), especially in the upper right part, the lower leftportion is still not sufficiently resolved. Figure 5(b)demonstrates that our modified method provides moresignificant improvement (R = 66.4%) and higher reso-lution, particularly in the lower left part, than does theGerchberg algorithm. We employed the average ofspectral magnitudes of three different prototype images(one of them is shown in Fig. 6) and incorporated thisinformation into the algorithm once at the tenth itera-tion cycle. It has been confirmed that the restored re-sults are not sensitive to the number of cycles in whichto insert the prototype information and are almost thesame if that number exceeds -5. The GSE curves forthe two algorithms are shown in Fig. 7, in which thesharp peak at the tenth cycle in the modified methodcorresponds to the incorporation of a priori informa-tion. Figure 8 shows the log power spectra of the re-stored images which clearly represents, in comparison

0 10 20 30

Fig. 8. Log power spectra of the restored images by using the originalGerchberg algorithm (- - -) and the modified algorithm with the single

replacement of a priori information (-).

with Fig. 4, that the spectrum is significantly extrapo-lated. Figures 7 and 8 suggest the superiority of themodified technique to the original Gerchberg algorithmin the simulated case.

Although the above comparison was made for thesame number of iterations, it may be more reasonableto compare the two algorithms at a more appropriatenumber of iterations. The optimal iteration numberk* is reached when the restoring coefficient R becomesthe largest. This coefficient begins to decrease there-after because of the ill-conditioned nature of the algo-rithm. In the above case k* is 124 for the Gerchbergalgorithm and 93 for the modified algorithm with Rs57.1 and 67.0%, respectively. The modified methodprovides considerable improvement in the rate of con-vergence toward k* as well as in the restored esti-mates.

As to the effects of noise, the restoring coefficientswith the optimal number of iterations for both algo-rithms at SNRs of 30, 25, 20, and 10 dB are shown inTable I. The inverse filter was used as Y(u,v) for 30-and 25-dB SNRs, and the Wiener filter was used for 20-and 10-dB SNRs. The rates of each R change based on


0 (

Table 1. Restored Results by Using the Gerchberg Algorithm and theModified One in the Presence of Noise

Gerchberg algorithm Modified algorithmSNR (dB) R(%)Ik* AR (%) R(%)/k* AR(%)

30 57.1/124 - 67.0/93 -25 49.2/35 13.8 60.4/22 9.920 41.3/22 27.7 54.1/12 19.310 35.9/13 37.1 47.9/12 28.5

1.0

0.

01 10 20 30 40 50

Fig. 9. GSE curves of the modified algorithm with the single re-placement (--- ) and with the triple replacement (-) of a priori

information.

0.

00 10 20 30

Fig. 10. Low power spectra of the restored images by using themodified algorithm with the single replacement (- --) and the triple

replacement (-) of a priori information.

its value at 30 dB are also shown in the table, AR = [(Rat 30 dB - each R)/R at 30 dB] X 100(%), which repre-sent the tendency of noise amplification of the algo-rithms. This table indicates that the modified methodalways demonstrates the larger restoring coefficientsand less amplification of noise than the Gerchberg al-gorithm.

Since the single incorporation of a priori informationresulted in desirable results, we then tried a multipleincorporation. The triple incorporation was imple-

mented, that is, the same replacement as before wasrepeated three times at the tenth, twentieth, and thir-tieth iteration cycles for 30-dB SNR. The GSE curvesand the low power spectra of the restored images areshown in Figs. 9 and 10, respectively, which show thegreater GSE and more appreciable extrapolation of thespectrum by the triple replacement in comparison withthe single replacement. The restored estimate of thetriple replacement has a slightly better appearance,especially in the lower left chart, than the single re-placement and has R = 72.2% at k* 94. These resultssuggest the availability of the multiple incorporationof a priori information.

We have been able to obtain satisfying results bymodifying the Gerchberg algorithm in our simulation.This may be partly due to the model object used, sincethe prototype images are easily determined for athree-bar chart. However, it may not be easy to findsuitable prototypes for more general objects. Hence theapplication of the modified superresolution algorithmto more practical restoration problems, such as astro-nomical imagery or electron microscopy, will have littledifficulty in selecting appropriate prototype images.

One probable way to choose reasonable prototypesis executing the original Gerchberg algorithm at firstand then examining its restored estimate. This re-stored image will enable one to determine the suitableprototypes which satisfy the two conditions describedin the previous section. Another difficulty is concernedwith the assumption that the extent of the object mustbe precisely known a priori. It is desirable to relax thisassumption because the rough estimate of the extentcan be extracted from the filtered image. Therefore,additional study is required about the prototype imagesand the extent of the object so that the modified tech-nique can be applied to the practical problems. Finally,we have considered only the rectangular pupil in thispaper, but extension to another pupil such as circularaperture is straightforward.

IV. Conclusion

We have described the digital restoration procedureof incoherent bandlimited images. The procedureconsists of two techniques: one is conventional spatialfrequency filtering and the other is the superresolutionalgorithm. The iterative algorithm of Gerchberg hasbeen used as the superresolution method. It has beendemonstrated that improved restored results could beobtained by incorporating a priori information basedon prototype images into the Gerchberg algorithm.The effectiveness of the proposed method has beenconfirmed through computer simulations. However,the practical application of the modified algorithm hasdifficult problems determining prototype images andthe extent of the object and is currently under investi-gation.

References1. D. Slepian and H. 0. Pollack, Bell Syst. Tech. J. 40, 43 (1961).2. J. L. Harris, Sr., J. Opt. Soc. Am. 54, 931 (1964).3. C. W. Barnes, J. Opt. Soc. Am. 56, 575 (1966).4. B. R. Frieden, J. Opt. Soc. Am. 57,1013 (1967).


5. C. K. Rushforth and R. W. Harris, J. Opt. Soc. Am. 58, 539(1968).

6. C. L. Rino, J. Opt. Soc. Am. 59, 547 (1969).7. H. Okuyama, Y. Ichioka, and T. Suzuki, Jpn. J. Appl. Phys. 15,

1087 (1976).8. R. W. Gerchberg, Opt. Acta 21, 709 (1974).9. A. Papoulis, IEEE Trans. Circuits Syst. CAS-22, 735 (1975).

10. M. S. Sabri and W. Steenaart, IEEE Trans. Circuits Syst. CAS-25, 74 (1978).

11. R. J. Marks II, Appl. Opt. 19, 1670 (1980).12. R. J. Marks II, Appl. Opt. 20, 1815 (1981).13. D. K. Smith and R. J. Marks II, Appl. Opt. 20, 2476 (1981).14. C. K. Rushforth and R. L. Frost, Proc. Soc. Photo-Opt. Instrum.

Eng. 231, 112 (1980).15. C. K. Rushforth and R. L. Frost, J. Opt. Soc. Am. 70, 1539

(1980).16. D. C. Youla, IEEE Trans. Circuits Syst. CAS-25, 694 (1978).

17. H. Stark, D. Cahana, and H. Webb, J. Opt. Soc. Am. 71, 635(1981).

18. D. Cahana and H. Stark, Appl. Opt. 20, 2780 (1981).19. H. Maitre, Opt. Acta 28, 973 (1981).20. J. W. Goodman, Introduction to Fourier Optics (McGraw-Hill,

New York, 1968), p. 118.21. P. DeSantis and F. Gori, Opt. Acta 22, 691 (1975).22. E. R. Cole, Computer Science Department, U. Utah, Salt Lake

City, UTEC-CSc-74-029 (1973).23. T. M. Cannon, Computer Science Department, U. Utah, Salt Lake

City, UTEC-CSc-74-091 (1974).24. T. G. Stockham, T. M. Cannon, and R. B. Ingebretsen, Proc. IEEE

63, 678 (1975).

25. A. V. Oppenheim and J. S. Lim, Proc. IEEE 69, 529 (1981).26. T. Honda, K. Kumagaya, and J. Tsujiuchi, Opt. Acta 24, 23

(1977).

TRANSFER OF COPYRIGHT AGREEMENT

Title of article:

Author(s):

The undersigned hereby assign(s) to the Optical Socity of America (the "Society") copyright ownership in the above-titledarticle, this assignment to be effective as of the date of acceptance of this article for publication in the above-named journaland to the extent transferable under applicable national law.

Authors (or their employers, in the case of works made for hire) reserve the following rights:(a) The right to use all or part of this article in future works of their own, including, but not limited to, lectures, lecture

notes, press releases, reviews, textbooks, or reprint books;

(b) The right to direct the Society to grant permission to third parties to republish all or part of the article or trans-lations thereof, provided, however, that without any limitation whatever on such right, the Society shall be entitled,at its option, to negotiate and to receive reasonable royalties in respect of any such republication;

(c) The right to refuse permission to third parties to republish all or pait of the article or translations thereof, with theexception of the permission contained in the copyright statement appearing in each issue of the journal.

The Society shall retain all proprietary rights with respect to journal issues as a whole.

Signature

Print name

Date Title

Signature

Print name

Title Date

Who Should Sign

This form must be signed by(1) each author who is not a U.S. Government employee, if reasonably practicable;and12) each such author's employer, if the article was prepared by such author(s) within the scope of his or her employ-

ment or was specifically ordered or commissioned by the employer.

if any author was a U.S. Government employee at the time this article was prepared, ad such article was prepared as part ofhis official duties, such author should sign the following form:

I hereby certify that this article was prepared by me as part of my official duties as an employee of theUnited States Government.

Signature Signature

Print name Print name

Date Title Title Date

This signed statement must be received by the editor's office before the article can be published. Address requests for further in-formation to the Executive Director, Optical Society of America, 1816 Jefferson Place, N.W., Washington, D.C. 20036 (telephone:202 223-8130).


Signature

Print name

Title

Signature

Print name

Title_ _

Documents

Digital restoration of incoherent bandlimited images