Download pdf - High-resolution Image Reconstruction

8/12/2019 High-resolution Image Reconstruction

1/4

HIGH-RESOLUTION IMAGE RECONSTRUCTION FROM LOWER-RESOLUTIONIMAGE SEQUENCES AND SPACE-VARYING IMAGE RESTORATION*

A. Murat TEKALP', Mehmet I


2/4

denote th e samples of tlie z f h frame of a sequence of lowresolution M x M ) mages of a continuous scene f ( z , y ,where a nd p are the relative displacements of the t t hframe in the z and y directions, respectively, with respectto a reference frame, A is the sampling grid spacing, h z , )is the PSF of the sensor, vt z,y)s the observation noisefor the z t h frame, denotes 2-D convolution, and b . , ) in-dicates the 2-D Kronecker delta function. We assume tha tf z,y) s bandlimited, but A is larger than the Nyquistsampling interval, so th at there is aliasing. Although thesensor PSF h z , ) acts more-or-less as an antialias filter,almost always some aliasing is present.

Now, assume that f k A j v , Z A ~ ) ,0 5 k,Z 5 A 1N > Af denote the samples of the continuous scene thatwould be obtained by a high resolution camera, such thatA N is the Nyquist sampl ing interval. The relation betweenthe images g, m,n ) and f k, (the independcnt variablesare changed for convenience) can be expressed as

N-1 N 1

k=O I = O

denotes the sensor PSF for t,he 2 frame, A denotes thearea of its argument, and Sh ., .) and Si(., ) denote thesupport of the high resolution and low resolution sensorscentered around the pixel ., .), respectively.

Then, the problem can be sta ted as: g i v e n N = L x M ,and a sequence of L 2 lo\v-resolut,ion images gi ,m, ) , o 5m , -< f 1, = 1,...L2,f a continuous scene f z , y ) ,obtain a higher-resolution N x A ) image f k, , 0 5 k, 5N 1. Tliis problem may arise under two different con-texts: (i) reconstruction of a high resolut.ion still-frameimage from its multiple low resolution recordings wherecontrolled (known) subpisel motion is introduced beforesuccessive exposures using high precision hardware, (ii) re-const.rnction of a higher resolution still-frame from a lowresolution video sequence under the assumption t hat thereis global (perhaps unknown) subpixel displacements be-tween the scene and the camera in successive frames (e.g.,due t o camera panning or imaging from a moving vehicle).In th e first case, i.e., when one has cont,rol over the sub-pixel shifts, i t can be easily seen tha t choosing t,hc dis-placenient,s to correspond to t he midpoint,s of th e low reso-lution sampling poiiits in the z and y directions and alongthe diagonal reduces the reconstruction problem to justdeconvolution of the sensor PSF. In thc latter case, t.hesubpisel displacements are usually unknown, and can beestimated using an ima.ge regist,ration algorit,hm, such asthe Fogel algorit.hm [8 ] or by subpixel-level block mat,chiiig[9]. In the case of video imagery, it is possible that onlya segment of the imagc is moving (space-varying motionproblem). Then, we recommend segmenting th e image iiit,oregions with uniform mot,ion, and apply the reconstructionrnet,Iiod separately over t.liese segments.

'The system of equations can be underdeteimGned if lessthan L 2 frames are available) or overdetermined (if more thanL2 frames are available).

2.2 Frequency Domain MethodIn thi s section we formulate the high-resolution image re-construction problem as a problem of solving a set of si-multaneous equ ations in the frequency domain. Taking theFourier transform of both sides of l ) ,we have

v , w r w 2 ) (4)where the quantities with capital let ters denote the Fouriertransforms of the respective signals. Given that g, m,n )undersamples f , z , y ) L times in both the and y direc-tions, there will be L 2 nonzero t,erms on t he RHS of (4) foreach frequency pair wI ; w2) . N o w that we have L2 suchequations, one from each frame i we can set up a set ofL 2 equations in as many unknowns at each frequency pairto recover the alias-free spect,rum F w 1 , wz).N o t es:

(i) The set of equations th at are formed at t he frequencypair (wL ~ ,2) are decoupled from the equations that areformed at another frequency pair. Thus, F w ,w2 an berecovered at each frequency pair independently.

(ii) The frequencies in th e range 0 5 tu1 , w2 5 shouldbe discretized to I< samples along each direction, where1 2 N , and a set of L2 equations in L2 unknowns will beformed for each sampled frequency pair.

(iii) The set of equations can only be solved in theleast squares sense due to the presence of observation noise

(iv) It is easy to see tha t if t,he image f z , y ) passedt.lirough a perfect antialias filter before the low resolutionsampling, there would be only one term on the RHS of(4), and recovery of a high resolution image would not bepossible no matter how many low resolution frames wereavailable. Thus, it is the aliasing terms in (4) that makethe recovery of a high resolution ima.ge possible.

This method constitutes a generalization of the tech-niques presented by Tsai and Huang [l], nd later by Kimet al. [IO] in that it includes the effects of both the sensorblurring and observation noise as well as th e aliasing effect.2.3 POCS MethodTh e high-resolution image reconstruct.ion problem can alsobe formulated as a problem of solving a set of simultaneousequations in t,he space domain. It is clear tha t from 2 ) wecan obtain A f x A equations for each frame, for a total ofL 2 x AI x N equat,ions in N x N unknowns, f k , . Giventhat N = L x A l , we have a set of simultaneous equa-tions in as many unknowns. IIowever, these equations areall coupled with each other, unlike the frequency domainformulat,ion. Further, in order for the equations to be lin-early independent all displacements between th e successiveframes must be at subpixel amounts. In 141, Frieden andAumann proposes a least squares solution for such a set ofequations.

K U 1 1 , UJ2).

111-170


3/4

Due to the large number of equations that have to besolved simultaneously, Irani and Peleg [6] proposed a solu-tion based on the principles of tomographic backprojection,whereas Stark and Oskoui [5] proposed a POCS formula-tion. However, both of these methods neglect th e effectof the observatio n noise. In this Section , we propose anextension of the POCS formulation [SI that includes theobservation noise. It should be noted that both th e leastsquares solution and the backprojection solution can beconsidered as special cases of the POCS solution.In order to account for the observation noise, we definethe following closed, coiivex constraints (one for each ob-served low resolution image pixel at each frame)

Cm,n;i = { f ( k , I : Irif) m, n)I 5 b o } ,0 5 m. n 5 h l 1, i = 1, . L z 5 )

where N 1 N-1r t f ) m , ) = g i m , n )- f k , I)hi m, n;k, , 6)

k O i = Oand 60 represents the confidence that we have in the ob-servation and is set equal to m u where au s the standarddeviation of the noise and c 2 0 is determined by an a ppropriate statistical confidence bound. Tlie quantity hi isas defined in 2-3).

The projection y k, ) = Pm,n;t[z R,)] of an arbitraryz k, ) onto C,,,,,,;i is defined as [I l l :Pm,n;t[r k, 111 =

Of course, in addit.ion, constraints such as finite energy,positivity, limited support can be utilized to improve theresults. This method is demonstrated by experimental re-sults in Section 4, where we have used relaxed projections.2.4 Intcrpolatioii-Restoration MethodIn this section, we propose an alternative two-step pro-cedure where the upsampling and the restoration of thesensor PSF are performed sequentially. Following, appro-priate registration of the L2 rames, the pixels on theseframes are mapped onto a single high resolution (N x N)grid on the basis of interframe displacement information.In general, when the displacements between the frames arenot controlled (pre-designed), t he pixels from these framesmay not exactly m ap onto t he high resolution samplinggrid points, and as a result the upsampled image becomesnonuniformly sampl ed. In order to arrive at a uniformlysampled upsampled image, we thu s need to perform inter-polation from these nonuniformly placed samples to thehigh-resolution uniform sampling grid points.

The problem of interpolation from nonuniformly sam-pled points to a uniformly sampled grid has been dis-cussed in the literature. Th e proposed mcthods includethose of thin-plate spline method of Franke [12], t,he it-erative method of Sauer and Allebach [], and the POCS

method of Yeh and Stark [13]. Of course, it is desirabletha t this interpola tion be error-free. lfowever, inevitab lysome interpolation errors are generally introduced in thisstep. Th e upsampling with the interpolation from nonuni-fornily spaced samples aims to undo the effect of under-sampling using t he additional samples from the neighbor-ing frames. If the interpolat ion were error-free, th e result-ing image would be de-aliased but would still suffer fromsensor blurring and noise. In our simulat ions, we performinterpolation using the local thin-plate spline method [12].Once the i mage is upsample d and de-aliased, we canaddress the restoration of the image at the higher sam-pling rate t o undo th e blurring of the sensor PSF. Restora-tion can be performed by any well-known deconvolutionmethod that takes the presence of noise into account, suchas Wiener filtering. T her e are two kinds of noise sources tobe considered a t this stage: (i) observati on noise of the sen-sor, and (ii) the interpol ation errors. Th e effects of thesenoise sources will be analyzed experimentally.

3. SPACE-VARYING IMAGE RESTORATIONIn this section, we propose the POC S formulation given inSection 2.3 as a new method for the restoration of space-variant blurred images. The POCS formulation presentedby Trussell and Civanlar [ll] for restoration of space-invariant images constrains the variance of the residualimage, and cannot be easily extended for space-variantrestoration since it involves inversion of huge matrices.Here, we consider each observed pixel value as a separateconstraint which yields a simpler computational algorithm.

Th e spatially variant blurred image can be representedas

N 1 N-I

k O 1=0where 0 5 m, n 5 N .

Comparing (8) with 2), clearly both the high resolutionimage reconstruction and the space-varying image restora-tion problems have the common formulation: N Z inear si-multaneous equations in N 2 nknowns, where the systemmatrix is not Toeplitz. Thus, the constraint sets and theirprojections given in Section 2.3 can be directly applied t oefficiently solve the space-variant restoration problem.

4. EXPERIMENTAL RESULTSSimulation results to reconstruct an image with 4 times theresolution of the imaging seiisor using a sequence of 16 lowresolution frames with two different approaches are shownin Fig. 1. A 256 x 256 aerial image is blurred wi th a 4 x 4uniform kernel, which simulates t he PS F of an imaging sen-sor, and white Gaussian noise of 20 dB s added t o simulateobservation noise. Th e 1 6 low-resolut.iort frames have beenobtained by subsampling the blurred image by a factor of4 (each frame assigned a different pixel of the 4 x 4 blocks).Fig. l a ) shows one such low-resolution ima ge interpolatedby 4 times for display purposes. Fig. l( b) is obtaine d by

111-17 1


4/4

registering the 16 low-resolution f rames using Fogels mo-tion estimation algorithm [8] and interpolating the imagevalues at high-resolution g rid points using Frankes thin-spline interpolator 1121. Note that this upsampled imagestill suffers from sensor blur. Fig. l(c ) shows the result ofthe two-step procedure (Section 2.4), and is obtained byapplying the Wiener filter to Fig. l(b ). Figs. l( d) and(e) are obtained by applying the proposed POC S based al-gorithm (Section 2.3) to the 16 low-resolution frames with60 = 0 and 60 = lu v, respectively. We note that Fig. l(e )suppresses the observation noise effectively as compared toFig. l (d ) (th e result of Stark-Oskoui method).

5. CONCLUSIONIt is interesting to note th at in the image restoration prob-lem the output image is at the same sampling rate as theinput image, and the resolution of the restored image islimited by that of the sensor P SF and t he sampling rate.On th e other han d, in the high resolution image reconstruc-tion problem, the outp ut image is at a higher sampling rateand resolution level than that of the image sensor. Th eresolution of the image can be increased by a factor of Lprovided th at there are L2 rames of the same scene avail-able, all recorded by shifting the camera at. subpixel levelswith respect to a reference frame. Our ability to decon-volve the sensor PSF may be limited by the zero-crossingsin the frequency response of the sensor.

In summary, the contributions of this paper are: (i) topresent different methods for high-resolution image recon-struction from a sequence of lower resolution images inthe presence of observation noise (Section 2) , and (ii) toshow that the POCS formulation proposed for the high-resolution image reconstruction problem can also be ap-plied to the space-varying still-frame image restorationproblem (Section 3).

References[I] R. Y. Tsai and T. S. Huang, Multiframe Image Restora-tion and Registration, in A d v a n c e s i n C o m p u t e r V i s i o nand Image Processing V o l . (T. S. Huang, ed.), pp. 317-

339, Greenwich, CT: Jai Press, 1984.[2] K.D. Sauer and J. P. Allebach, Iterative Reconstructionof Band-limited Images from non-uniformly Spaced Sam-ples, IEEE Trans . Circui t s Sys t . , vol. CAS-34, pp. 1497-

1505, 1987.[3] IC. Aizawa, T. Komatsu, and T . Saito, Acquisition of VeryHigh Resolution Images Using Stereo Cameras, in P T O C .SPIE Visual Communicat ions and Image Process ing 91,(Boston, Massachusetts), pp. 318-328, November 1991.[4] B. R. Frieden and II. H.C. Au nm n, Image Reconstmc-tion from Multiple 1-D Scans Using Filte red Localized Pro-jection, Appl ied Opt ics , vol. 26, pp. 3615-3621, Septem-ber. 1987.

[6] M. Irani and S. Peleg, Improving Resolution by ImageRegistration, CV GIP : Graphical hfode ls and Image PT OC. ,vol. 53, pp. 231-239, May, 1991.[7] M. I. Sezan and A. M. Tekalp, Adaptive Image Restora-tion with Artifact Suppression Using the Theory of Con-vex Projections, IEE E Tr ans. Acoust. , Speech, and SignalPTO C . , ol. ASSP-38, pp. 181-185, January, 1990.[SI S. V. Fogel, Estimation of Velocity Vector Field fromTime-Varying Image Sequences, C o m p u t . V i s i o n G r a p h -ics Image Process. : Image Unders tanding, vol. 53, pp. 253-

287, May 1991.[9] M. Bierling, Displacement Estimation by HierarchicalBlockmatclung, in PTO C .SPIE Visual Communicat ionsand Image Processing 88, pp. 942-951,1988.[lo] S. P. Kim, N. I