Real-Time Imaging 9 (2003) 61–72

Real-time iterative framework of regularized image restorationand its application to video enhancement

Sungjin Kima, Jeongho Shina,b, Joonki Paika,*aDepartment of Image Engineering, The Graduate School of Imaging Science, Chung-Ang University, 221 Huksuk-Dong, Tongjak-Ku,

Seoul 156-756, South Koreab Information and Electronics Program, Korea Institute of S&T Evaluation and Planning, 275 Yangjae-Dong, Seocho-Ku, Seoul, South Korea

Received 29 April 2002; received in revised form 6 November 2002; accepted 7 January 2003

Abstract

A novel framework of real-time video enhancement is proposed. The proposed framework is based on the regularized iterative

image restoration algorithm, which iteratively removes degradation effects under a priori constraints. Although regularized iterative

image restoration is proven to be a successful technique in restoring degraded images, its application is limited within still images or

off-line video enhancement because of its iterative structure. In order to enable this iterative restoration algorithm to enhance the

quality of video in real-time, each frame of video is considered as the constant input and the processed previous frame is considered

as the previous iterative solution. This modification is valid only when the input of the iteration, that is each frame, remains

unchanged throughout the iteration procedure. Because every frame of general video sequence is different from each other, each

frame is segmented into two regions: still background and moving objects. These two regions are processed differently by using a

segmentation-based spatially adaptive restoration and a background generation algorithms. Experimental results show that the

proposed real-time restoration algorithm can enhance the input video much better than simple filtering techniques. The proposed

framework enables real-time video enhancement at the cost of image quality only in the moving object area of dynamic shots, which

is relatively insensitive to the human visual system.

r 2003 Elsevier Science Ltd. All rights reserved.

Keywords: Real-time image processing; Image restoration; Interpolation; Video enhancement; Regularization

1. Introduction

Image and video enhancement plays an importantrole in multimedia communication and its applicationareas. Throughout the decades, many researchers havestruggled to maximize the quality of image data withminimum cost of communication and storage. Theirefforts were realized, to a certain degree, by variousimage and video compression technologies [1–5] whichtry to overcome the existing upper limit of image qualitydefined by the classical communication theory [6]. Fastgrowing need to multimedia communication overrelatively fixed communication channels and storages,however, results in further degradation in compressed

images, and therefore image and video enhancementbecomes necessary to provide an acceptable qualityimage under compression. Among various enhancementtechniques, blocking artifact removal and image inter-polation are the most important because their perfor-mances directly affect quality of images and they usuallyrequire high computational complexity.

Regularized image restoration has been proposed inthe literature to remove undesired artifact from thedegraded images, and shown successful results inrestoring degraded images corrupted by simple, space-invariant artifacts, such as: additive noise, motion blur,and defocusing [7]. Recently, regularized image restora-tion algorithms have been modified and applied toremove blocking artifacts caused by standard compres-sion techniques [8–13] and to interpolate subsampledimages [14–21].

In spite of successful application of regularizedimage restoration to enhance compressed images, the

*Corresponding author. Department of Image Engineering, Grad-

uate School of Advanced Imaging Science, Chung-Ang University,

Seoul 156-756, South Korea.

E-mail address: paikj@cau.ac.kr (J. Paik).

1077-2014/03/$ - see front matter r 2003 Elsevier Science Ltd. All rights reserved.

doi:10.1016/S1077-2014(03)00003-2

previously mentioned restoration algorithms were de-vised mainly for processing still images and off-linevideo. This restriction originates from the iterativestructure of the regularized restoration algorithm. Morespecifically, general regularized restoration algorithms,given an initial guess, iteratively refine the solution untilthe convergence condition is satisfied. Since the proces-sing time of one iteration is, in general, equivalent to theinterval of temporarily adjacent frames in a video data,multiple iterations results in delay of the same numberof frames, which makes it impossible to apply aregularized restoration algorithm to real-time videoenhancement. A fast image restoration algorithm usingmodified Hopfield network was proposed in [22], andregularization-based fast block artifact removing algo-rithms were proposed in [23,24].

In this paper, we propose a real-time framework ofregularized iterative image restoration, and its applica-tions to compressed video for block artifact removal andinterpolation. The proposed framework is based on theassumption that each frame can be decomposed intostationary background and moving objects. Thisassumption is valid, for example, for video telephonyand video surveillance systems with a stationary camera.A video obtained from a dynamic camera does notsatisfy this assumption, but the dynamic backgroundcan be made stationary by suitable camera calibrationand digital image stabilization techniques [25,26]. Wefurther assume that such decomposition can beperformed at one of the following levels:

Pixel-level: This is the ideal level of decomposition. Apixel-based moving object extraction can be realized bythe optical flow equation [27]. For example, objectextraction from a video object plane (VOP) in MPEG-4falls into this category [4].

Block-level: If we extract a moving region at theblock-level, computational complexity is significantlyreduced at the cost of accuracy.

Frame-level: This level results in the simplest decom-position algorithm. If we decompose an entire videosequence into two sets of frames, called shots, with andwithout motion as shown in Fig. 1, we can adaptively

apply the existing regularized restoration algorithm toshots without motion and a simple closed-form filter toshots with motion.

The pixel-based moving object extraction techniqueis out of the scope of this paper, and the cur-rently available technique requires so much com-putation that real-time implementation is impossible.Instead the proposed real-time enhancement frame-work is based on either block-level or frame-leveldecomposition.

The paper is organized as follows. A brief review ofregularized image restoration is given in Section 2,which includes the mathematical model of imagedegradation and the general form of an iterativealgorithm. In Section 3 a block-level moving objectextraction algorithm is proposed for stationary back-ground generation, and in Section 4 the proposed real-time restoration framework is described. Application ofthe proposed framework to enhancement of low-resolution video is provided in Section 5. Experimentalresults and conclusions are given in Sections 6 and 7,respectively.

2. Regularized iterative image restoration: a review

Image degradation is unavoidable in most imageprocessing systems. Typical image degradation includesmotion blur, defocusing, low-resolution by subsam-pling, and blocking artifact resulting from imagecompression, to name a few. Image restoration is toremove such degradation and recover the higher qualityimage as close to the original image as possible [28].

The regularized iterative image restoration algorithmis particularly suitable for removing such degradation,because: (i) this process minimizes a user-definedobjective function, which represents error between thedesired solution and the current estimate, (ii) thisalgorithm can incorporate various types of prior knowl-edge about the desired solution, which is usuallyimplemented in the form of constraints, and (iii) thisalgorithm can also be implemented in an accelerated

Fig. 1. A sample video sequence decomposed into shots with and without motion. yi represents the ith frame of the video.

S. Kim et al. / Real-Time Imaging 9 (2003) 61–7262

manner by pre-conditioning the iteration [29]. Regular-ized restoration is based on the general formulation ofregularized optimization, which can be expressed as

argminx f ðxÞ ¼ gðxÞ þ lhðxÞ ð1Þ

where x represents the lexicographically ordered vectorof the image. The first term in the objective function,gðxÞ ¼ jjy � Hxjj represents the data compatibility term,where y and H; respectively represent the degradedimage and a block matrix that models degradation. Thesecond term, hðxÞ; represents the regularizing term thatincorporates prior knowledge about the solution, and lis the regularization parameter that controls the ratio ofminimization between gðxÞ and hðxÞ [7,29].

2.1. Image degradation model

In order to fit the general regularized optimizationproblem, given in (1), to the practical image restorationproblem, the mathematical model of degradation mustbe defined first. Assume that a L � L digital image isgiven. Let x be the lexicographically ordered L2 � 1vector, which represents the original image. H is a L2 �L2 matrix, which represents the degradation system. Thedegraded image can then be expressed as

y ¼ Hx þ Z ð2Þ

where Z represents additive noise. This image degrada-tion model is illustrated by a block diagram in Fig. 2.

2.2. Regularized iterative restoration

If we neglect additive noise Z in (2), the imagedegradation model can be simplified as

y ¼ Hx ð3Þ

In order to estimate the original image x from thesimplified image degradation model in (3), we need tocompute the inverse of H and apply the inverse to thedegraded image y as

x ¼ H�1y ð4Þ

This approach, however, has two major problems: (i)the noise effect cannot be neglected in most practicalimage processing applications, and (ii) the degradationsystem H is most probably ill-conditioned, and therefore

the estimation in (4) becomes an ill-posed problem. Inorder to solve such ill-posed problems, an iterativeapproximation method has been used as

xðkþ1Þ ¼ FxðkÞ ð5Þ

where F represents the iteration operator which makesthe kth iterative solution, xðkÞ; move closer to the desiredsolution that minimizes the functional given in (1).Finally, the kth iteration of regularized restorationalgorithm is given as

xðkþ1Þ ¼ xðkÞ þ b½HT y � ðHT H þ lCT CÞxðkÞ ð6Þ

where C represents a high pass filter, which serves as apriori smoothness constraint. Parameters l and b arerespectively called the regularization parameter and theconvergence rate control parameter. More detail for theregularized restoration algorithm can be found in [7,30].

3. Background generation using a fixed camera

The background can be generated by dividingan image frame into several regions. Since we assumethat majority of motions occur in the horizontaldirection, horizontal segmentation is considered asshown in Fig. 3. This assumption is valid for video

x H

Additive Noise,�

DegradationSystem

Original Image

G

RestorationProcess

x̂

Restored Image

Fig. 2. Block diagram of the image degradation model and the image restoration process.

Time

Background

Frame 1 Frame 2 Frame k

. . . . . .. . .. . .

. . .

Fig. 3. Illustration of the background generation algorithm. Input

image frames are shown in the upper row, and the generated

background image is shown below. A small dark block in input

frames represents a moving block.

S. Kim et al. / Real-Time Imaging 9 (2003) 61–72 63

telephony and video surveillance systems, but theextension of segmentation in both horizontal andvertical directions is straightforward. We also assumethat each frame of the input video sequence is similar tothe previous ones. In other words, there is no significantdifference between the current and the previous framesexcept specific regions with moving objects, and thestationary region without motion is defined as thebackground.

For motion detection, we regard the previous frameas the reference frame, which is compared with thecurrent frame. Each block of the reference frame iscompared with that of the current frame. The typicalsize of blocks is 16� 16. Sum of absolute difference(SAD) is used as a measure of difference betweentwo blocks. Let Bði; jÞ represent the set of pixels in theði; jÞth block, SAD of the corresponding block is thendefined as

SADði; jÞ ¼X

ði;jÞABði;jÞ

jyðk�1Þði; jÞ � yðkÞði; jÞj ð7Þ

where yðk�1Þ and yðkÞ respectively represent the previousand the current image frames. If the SAD becomeslarger than a pre-specified threshold, the correspondingblock is determined to have motion.

The proposed background generation algorithm isillustrated in Fig. 3. Without loss of generality, assumethat an object is moving in the horizontal direction.Each input frame is horizontally segmented into thebackground and moving regions. The background of theframe updates the corresponding part of the back-ground image, which will be completely filled andupdated by the following frames.

4. Real-time iterative framework of regularized

restoration

Given an image degradation model such as (2),the general image restoration process based onthe regularized constrained optimization approach isto find the minimum solution of an objective functiongiven as

f ðxÞ ¼ jjy � Hxjj2 þ ljjCxjj2 ð8Þ

where C and l; respectively represent a high passfilter for incorporating a priori smoothness con-straint and the regularization parameter that controlsthe fidelity to the observed data and smoothness ofthe restored image. To find the solution for minimizingthe objective function given in (8), we apply thefollowing iterative method,

xðkþ1Þ ¼ xðkÞ þ b½HT y � ðHT H þ lCT CÞxðkÞ ð9Þ

where xðkþ1Þ and b respectively represent the restoredimage after ðk þ 1Þ th iterative solution and the steplength which controls the convergence rate.

4.1. Real-time framework for a motionless video shot

If we use the original regularized restoration algo-rithm in (9), which is assumed to converge after M

iterations, for enhancing a video, each frame delays byM frames. If the video consists of N frames, total delaybecomes MN frames, which disables real-time proces-sing. Fig. 4 illustrates how frame delay occurs when theoriginal iterative regularized restoration algorithm isused.

The restored ith frame, denoted by #xi; for i ¼ 1;y;N ;as shown in Fig. 4, is obtained as

xð0Þi ¼ HT yi

xðkþ1Þi ¼ x

ðkÞi þ b½HT yi � ðHT H þ lCT CÞxðkÞ

i

for k ¼ 0; 1;y;M � 1; and #xi ¼ xðMÞi ð10Þ

where subscripts represent the frame number andsuperscripts represent the number of iterations. Assumethat a shot of the input video sequence has little motion,so that N input frames in the motionless shot areapproximately the same, such as

y1Ey2E?EyN ð11Þ

Based on (11), the first frame is used as the initial guessof the regularized iteration as

xð0Þ1 ¼ HT y1

#x1 ¼ xð0Þ1 ð12Þ

y1 yN _ 2

x 0 11

x Mx102x 1

2x Mx2

x̂1ˆ ˆ ˆ ˆ ˆ

y2 y3 yN _ 1

yN

1

... ...

... ...

...

x2 x3 xN _ 2

xN _ 1

xN

....

Fig. 4. Frame delays occurred in a shot with N frames, when an

iterative algorithm is used.

S. Kim et al. / Real-Time Imaging 9 (2003) 61–7264

and the #xi for i ¼ 2;y;N; is updated as

xð1Þ1 ¼ x

ð0Þ1 þ b½HT y1 � ðHT H þ lCT CÞxð0Þ

1

#x2 ¼ xð1Þ1

^

xðN�1Þ1 ¼ x

ðN�2Þ1 þ b½HT y1 � ðHT H þ lCT CÞxðN�2Þ

1

#xN ¼ xðN�1Þ1

ð13Þ

In order to implement a real-time restoration algorithm,Eq. (13) can be rewritten as

xð1Þ1 ¼ bHT y1 þ b½I � ðHT H þ lCT CÞxð0Þ

1

#x2 ¼ xð1Þ1

^

xðN�1Þ1 ¼ bHT y1 þ b½I � ðHT H þ lCT CÞxðN�2Þ

1

#xN ¼ xðN�1Þ1

ð14Þ

Since H and C respectively represents the degradationand a priori smoothness constraint matrices, both ofwhich are given before the iteration starts.

The proposed iteration scheme for a motionlessshot is illustrated in Fig. 5. As shown in Fig. 5, N

input frames in the motionless shot are utilized asinput images of the proposed real-time iterative re-storation. By assuming that input image framesare approximately the same as y1; the iterative restora-tion formulation is greatly simplified. Moreover, theith restored image frame, #xi can be calculated by asimple filtering operation which can be represented bymultiplication of the pre-defined matrix I � ðHT H þlCT CÞ and the previous solution vector x

ðkÞ1 given in

(14). In other words, each frame can be restored by asingle filtering operation, which enables real-time videoenhancement.

4.2. Decomposition-based real-time regularized

restoration

In general, we can hardly find exactly the sameframes in a video, which means the frame-leveldecomposition is almost impractical. For the blockand pixel-level decompositions, a necessary conditionfor the proposed real-time framework is that thevideo must be captured by a stationary camera.This condition ensures that the stationary backgroundof each frame is the same. In the video capturedby a fixed camera the background region can beenhanced by the proposed regularized iteration andthe moving region is either enhanced by simplefiltering or remains unprocessed. The proposedregion-adaptive real-time restoration consists oftwo steps. First, we extract the background usingthe method described in the previous section, andthen the stationary background is enhanced byusing the proposed restoration method. Second, therestored background and the moving regions are merged(Fig. 6).

Since the extracted background from a fixed camera isthe same, we have

yb1¼ yb2

¼ y ¼ ybNð15Þ

where ybNrepresents the Nth reconstructed background.

Since (15) is equivalent to (11), (12) and (14) can berespectively rewritten as

xð0Þb1

¼ HT yb1

#xb1¼ x

ð0Þb1

ð16Þ

and

xð1Þb1

¼ bHT yb1þ b½I � ðHT H þ lCT CÞxð0Þ

b1

#xb2¼ x

ð1Þb1

^

xðN�1Þb1

¼ bHT yb1þ b½I � ðHT H þ lCT CÞxðN�2Þ

b1

#xbN¼ x

ðN�1Þb1

ð17Þ

The proposed algorithm is summarized as follows.

Algorithm 1. Decomposition-based real-time regular-ized restoration

Step 1: Generate the background image using the firstT input frames, as described in Section 3. After T

frames pass, go to the next step.Step 2: Extract moving regions at each frame by

comparing with the previously generated backgroundimage.

Step 3: Apply the real-time regularized restorationprocedure given in (16) and (17) to the background anda simple filter to the moving region.

Step 4: Combine the restored background and thefiltered moving region.

y1 y2 = y1

01

x 11

x 21

x1N _1x

ˆ

y3 = y1 yN = y1

x1x̂2

x̂3 x̂N

. . .

. . .

. . .

Fig. 5. The real-time iterative restoration scheme for a motionless

shot.

S. Kim et al. / Real-Time Imaging 9 (2003) 61–72 65

5. Enhancement of low-resolution video: an application

Among various types of degradation, compressionand subsampling are the major factors that degrade thequality of video. Compression is usually necessary forvideo communication because of the limited bandwidthof the communication channel. Subsampling is anotherway to reduce the amount of video data and computa-tional overhead at the cost of resolution. Bothcompression and subsampling can be mathematicallymodeled, and therefore their degradation effects can beremoved by the regularized iterative restoration algo-rithm. Mathematical model of degradation resultingfrom video compression and the corresponding restora-tion algorithms has also been proposed in Refs. [23,24].In this section, we present the image degradation modelresulting from combined compression and subsamplingand the corresponding restoration results using theproposed real-time framework with particular emphasison the video enhancement aspect.

High-resolution (HR) image restoration refers torestoration of a low-resolution (LR) image by removingdegradation due to subsampling. Application areas ofHR image restoration include, but are not limited to:digital high-definition television (HDTV), aerial photo-graphy, medical imaging, video surveillance, and remotesensing [31].

Many algorithms have been proposed to obtain anHR image from one or more LR images. Conventionalinterpolation algorithms, such as zero-order or nearestneighbor, bilinear, cubic B-spline, and the DFT-basedinterpolation, can be classified by a basis function usedfor signal synthesis [32–34]. Since these algorithms focus

on just enlargement of an image without considerationof the degradation factor, they cannot restore theoriginal HR image. In order to improve the performanceof the previously mentioned algorithms, a spatiallyadaptive cubic interpolation method has been proposedin [35]. It is well-known that image interpolation is anill-posed problem. More specifically, sub-samplingprocess can be regarded as a general image degradationprocess. Therefore, the regularized image restorationalgorithm can successfully find the inverse solutiondefined by the subsampling process with a prioriconstraint [36,37]. Many regularization-based or similarinterpolation methods have been proposed in Refs. [38–41].

5.1. Image degradation model for combined subsampling

and compression

Enlargement of compressed video has wide applica-tion areas. For example, when we look at a JPEG codedpicture by using an image viewer, we can selectivelyenlarge or reduce the size of the picture by interpolationand subsampling, respectively. Consider an LR image ofsize M � N ; which can be obtained by subsampling theoriginal pM � pN image x: This LR image is thencompressed by block discrete cosine transform (BDCT).The original pM � pN image can be reconstructed fromthe subsampled-compressed image by regularized imagerestoration. For the reconstruction or enhancement weneed to define the image degradation model for bothsubsampling and compression.

Fig. 7 shows the combined subsampling-compressiondegradation process, where HS represents subsampling,

Frame

Background

Object

Result

ˆ

backgroundreconstruction

+ +

ˆ ˆ

o1oN _1

xb1 + o1

ˆ ˆ ˆ

xb2 + o2

xbN _1

+oN _1

xbN

+oN

ˆxb2

+o2x

b1+o

1

o2 oN

xb1 x̂b2x

bN _1 xbN

xbN _1

+oN _1

xbN

+oN

+ +

. . .. . .

. . .

. . .. . .

. . .. . .

Fig. 6. Background generation, moving object extraction, and combination of restored background and filtered object.

S. Kim et al. / Real-Time Imaging 9 (2003) 61–7266

C and C�1 respectively represent the forward and theinverse DCTs, and Q and D�1 respectively represent thequantization and inverse quantization matrices [23].

Based on the degradation process shown in Fig. 7, thecorresponding mathematical model is given as

H ¼ HCHS ð18Þ

where HC ¼ C�1D�1QC represents the degradationmatrix due to compression. x; HS and HC respectivelyrepresent the lexicographically ordered, p2MN � 1;MN � p2MN; and MN � MN vectors. Detailed de-scription of each degradation model can be found in[21,23].

5.2. Real-time restoration of subsampled-compressed

video

The proposed real-time framework of regularizedrestoration was described in (16) and (17). Afterobtaining the image degradation model H and a priorismoothness constraint C; application of the restorationprocess is straightforward.

6. Experimental results

In this section we present two experimental results.The first experiment is made to justify the proposedimage degradation model for combined subsampling-compression, given in Section 5, and the secondexperiment shows the test results of the proposed real-time video restoration algorithm.

6.1. Image degradation model for combined subsampling-

compression

For evaluation of the proposed image degradationmodel we used 256� 256 pepper image. The originalimage was first subsampled by the factor of 2, and thencompressed with a JPEG quantization table shown inFig. 8. This subsampled-compressed image has lowerresolution than the original image and blocking artifactsat the boundary of the BDCT as shown in Fig. 9(a). ThisLR image was used as an input of the restoration

algorithm. Fig. 9(b) shows the two times magnifiedversion of the LR image with restoration for removingonly subsampling degradation. Since restoration doesnot deal with the degradation by compression, blockingartifacts remains and are noticeable in Fig. 9(b).Fig. 9(c) shows the two times magnified version of theLR image with restoration for removing the combinedsubsampling-compression degradation. The restoredimage has significantly reduced blocking artifacts byconsideration of subsampling-compression degradationwith l ¼ 10 and b ¼ 0:05: Finally, Fig. 9(d) shows thetwo times magnified version of the LR image withrestoration for removing the combined subsampling-compression degradation with l ¼ 1 and b ¼ 0:05:

For objective comparison of two different degrada-tion models, we computed peak-to-peak signal-to-noiseratio (PSNR) as

PSNR½dB ¼ 10 log10

L2 � 2552

jjx � #xjj2ð19Þ

where x represents the original HR image, #x the restoredimage, and L the size of the image. PSNR values of twodifferent image restoration results versus the number ofiteration are shown in Fig. 10. As shown in Fig. 10, therestoration results considering combined subsampling-compression degradation gives better performance byapproximately 0.19 dB. Based on both subjective andobjective evaluation, the proposed subsampling-com-pression degradation model gives better restorationperformance.

6.2. Real-time video enhancement

In order to test the proposed real-time restorationalgorithm, we used a video sequence of size 320� 240captured by a fixed-view surveillance camera. One frameof the video sequence is shown in Fig. 11. This video

Fig. 8. Quantization table used for JPEG-based compression.

xy

Compression

C Q D -1

Subsampling Interpolation

HS

Degradation Process

x̂

C -1

Fig. 7. Image degradation process for combined subsampling-com-

pression.

S. Kim et al. / Real-Time Imaging 9 (2003) 61–72 67

sequence is appropriate for the experiment because thecamera was fixed and the localized target object caneasily be extracted from the stationary background.According to the combined subsampling-compression

Fig. 9. (a) The input LR image made by two times subsampling and JPEG compression, (b) The restored image using only subsampling degradation

model with l ¼ 10 and b ¼ 0:05 (PSNR ¼ 25:00 ðdBÞ), (c) The restored image using the combined subsampling-compression degradation model with

l ¼ 10 and b ¼ 0:05 (PSNR ¼ 25:19 ðdBÞ), and (d) The restored image using the combined subsampling-compression degradation model with l ¼ 1

and b ¼ 0:05 (PSNR ¼ 24:78 ðdBÞ).

Fig. 11. One frame of the original video sequence captured by a fixed-

view surveillance camera. Note that the stationary background is

dominant throughout the image, and the moving person is localized.

Fig. 10. Comparison of two different degradation models in the sense

of PSNR: The dotted curve represents the restoration results using

only subsampling degradation model as shown in Fig. 9(b), and the

solid curve represents the results using the combined subsampling-

compression degradation model as shown in Fig. 9(c).

S. Kim et al. / Real-Time Imaging 9 (2003) 61–7268

degradation model in (18), the two times subsampledand compressed version of the original video was usedas an input to the proposed real-time restorationalgorithm.

Based on Algorithm 1, the background image wasgenerated first from the video. The background imageconsists of four evenly spaced vertical regions as shownin Fig. 3. The first and the second regions were obtained

from the fourth and sixth frame, respectively. The thirdand the fourth regions were obtained from the 17th andthe 38th frames, respectively. The background genera-tion process is illustrated in Fig. 12. In other words, thebackground image was completely generated after the38th frame, and then the rest part of Algorithm 1 can beperformed.

After the LR background image is generated, it mustbe enlarged to obtain the original HR backgroundimage. Two differently enlarged background images areshown in Figs. 13 and 14. Fig. 13(a) shows the two times

Fig. 12. The background generation process. Background images with: (a) the first region from the fourth frame, (b) the first and second regions (the

second region is obtained from the sixth frame), (c) the first, second, and third regions (the third region is obtained from the 17th frame), and (d) all

four regions (the fourth region from the 38th frame.).

Fig. 13. The enlarged 70th background frames by using (a) the zero-order interpolation algorithm (PSNR ¼ 19:16 dB) and (b) the bicubic

interpolation algorithm (PSNR ¼ 18:91 dB).

Fig. 14. The enlarged 70th background frame by using the proposed

real-time regularized restoration algorithm. (PSNR ¼ 19:53 dB).

Fig. 15. The moving region extracted from the 87th frame.

S. Kim et al. / Real-Time Imaging 9 (2003) 61–72 69

enlarged images using simple zero-order interpolationalgorithm. Fig. 13(b) shows the two times enlargedimages using the bicubic interpolation algorithm. Due tothe nature of the interpolation algorithm, the enlargedimage shows blocking artifacts and does not haveenhanced resolution. On the other hand Fig. 14 showsthe 70th background frame, which is enhanced by usingthe proposed real-time regularized restoration algo-rithm. This background image is obtained after 32iterations, and is shown 0.37 dB improvement overFig. 13(a). If we keep iterating the background imageshown in Fig. 14, both the subjective quality and thePSNR value also keep increasing.

After generating and restoring the background image,the moving region is to be extracted from the inputframe. The extracted moving region from the 87th frameis shown in Figs. 15–17, and the combined restoredbackground and filtered moving region is shown in

Fig. 17. If we compare Fig. 17 with the enlarged imagewithout restoration shown in Fig. 16(a) and (b), therestored image gives approximately 0.25 dB improve-ment in PSNR. The proposed real-time video enhance-ment algorithm is compared with simple enlargementmethod in the sense of PSNR versus the number ofiterations, as shown in Fig. 18. Until the 38th frame norestoration process occurs, and therefore two methodsgive the same result. After the background image isgenerated at the 38th frame, the real-time restorationresult outperforms simple enlargement method. Finally,we also added another set of experiment as shown inFig. 19(a) and (b).

To ensure the real-time processing, we computed theprocessing time of the proposed algorithm. The 100frame 320� 240 video sequence was processed by usinginteractive data language (IDL) in a Pentium 4 1GHzpersonal computer. The proposed real-time restorationalgorithm took 84 s to process 100 frames, while theoriginal form of iterative restoration algorithm took495 s.

Fig. 16. The enlarged 87th frames by using (a) the zero-order interpolation algorithm (PSNR ¼ 19:30 dB) and (b) the bicubic interpolation

algorithm. (PSNR ¼ 19:06 dB).

Fig. 17. The combined restored background and filtered moving

region. (PSNR ¼ 19:53 dB).

Fig. 18. Comparison of two different enlarged sequences in the sense

of PSNR. The black curve represents the enhanced sequence by using

the proposed real-time restoration, and the dotted curve represents just

enlarged sequence without restoration. Until the 38th frame, two

sequences are the same because no restoration process occurs.

S. Kim et al. / Real-Time Imaging 9 (2003) 61–7270

7. Conclusions

We proposed a real-time framework for regularizediteration and its application to restoration of sub-sampled-compressed degradation. In order to restore asubsampled-compressed image, the combined imagedegradation model was proposed. The proposed real-time framework first generates the background image,extracts the moving region by subtracting the back-ground image from the current frame, and finallycombines the restored background and the filteredmoving region. The proposed restoration frameworkcan efficiently enhance both quality and resolution ofvideo in real-time at the cost of slight performancedegradation especially in the moving region, which isrelatively insensitive to the human visual system.

Our contribution can be summarized as:

(i) to analyze the structure of restoration,(ii) to decompose each iteration step into stationary

and dynamic parts, which are respectively denotedby the first and second terms in the right side ofEq. (14),

(iii) to segment each frame into background andmoving regions, which respectively correspond tothe stationary and dynamic parts of the decom-posed iteration step.

Extensive experiment was conducted to demonstratethe feasibility of the proposed image degradation modeland the real-time restoration framework. Experimentalresults show that the combined subsampling-compres-sion degradation model outperformed the only subsam-pling degradation model in restoring the compressedvideo. Each step of the proposed real-time restorationalgorithm was tested and the corresponding results werepresented.

Application of the proposed real-time framework isrestricted to surveillance or the equivalent quality video

systems because of the assumption that the input videomust be captured by a fixed-view camera. However, ifthe background image generation method is modified todeal with panned or tilted video input, the proposedalgorithm can be used to extended application areas.

References

[1] JPEG, http://www.jpeg.org.

[2] ITU-T Recommendation H.263: Video Coding for Low Rate

Communication, February 1998.

[3] MPEG, http://mpeg.telecomitalialab.com.

[4] MPEG-4 Video Verification Model, ver. 15.1, JTC1/SC29/WG11

5829, 2000.

[5] Mitchell JL, Pennebaker WB, Fogg CE, LeGall DJ. MPEG video

compression standard. London: Chapman and Hall, 1996.

[6] Shannon CE, A mathematical theory of communication, Bell

System Technical Journal 1948; 27: 379–423, 623–56.

[7] Katsaggelos AK. Iterative image restoration algorithm. Optical

Engineering 1989;28(7):735–48.

[8] Yang Y, Galatsanos NP, Katsaggelos AK. Regularized recon-

struction to reduce blocking artifacts of block discrete cosine

transform compressed images. IEEE Transactions on Circuits and

Systems for Video Technology 1993;3(6):421–32.

[9] Yang Y, Galatsanos NP, Katsaggelos AK. Projection-based

spatially adaptive reconstruction of block transformed com-

pressed images. IEEE Transactions on Image Processing

1995;4(7):896–908.

[10] Stevenson R, Reduction of coding artifacts in low-bit rate video

coding. Proceedings of the 38th Midwest Symposium, Circuits,

Systems, August 1995. p. 854–7.

[11] O’Rourke T, Stevenson R. Improved image decompression for

reduced transform coding artifact. IEEE Transactions on Circuits

and Systems on Video Technology 1995;5(6):490–9.

[12] Joung SC, Paik JK, Modified regularized image restoration for

post processing inter-frame coded images. Proceedings of the 1999

International Conference. Image Processing, vol., 3, October

1999. p. 474–8.

[13] Joung SC, Kim SJ, Paik JK, Postprocessing of inter-frame coded

images based on convex projection and regularization. Proceed-

ings of the 2000 SPIE Image, Video Communication, Processing,

vol. 3974, 2000. p. 396–404.

Fig. 19. The restored image frame by using (a) the zero-order interpolation algorithm (PSNR ¼ 21:54 dB) and (b) the combined subsampling-

compression degradation model. (PSNR ¼ 22:03 dB).

S. Kim et al. / Real-Time Imaging 9 (2003) 61–72 71

[14] Schultz RR, Stevenson RL. A bayesian approach to image

expansion for improved definition. IEEE Transactions on Image

Processing. 1994;3:233–42.

[15] Patti A, Sezan MI, Tekalp AM, High-resolution image recon-

struction from a low-resolution image sequence in the presence of

time varying motion blur, Proceedings of the 1994 International

Conference on Image Processing, November 1994, Austin, Texas.

[16] Patti AJ, Sezan MI, Tekalp AM, High-resolution standards

conversion of low resolution video, Proceedings of the 1995

International Conference on Acoustics, Speech, Signal Processing,

Detroit, MI, 1995. p. 2197–200.

[17] Tom BC, Katsaggelos AK, An iterative algorithm for improving

the resolution of video sequences. Proceedings of the SPIE Visual

Communications, Image Proceedings, Orlando, FL, March 1996.

p. 1430–8.

[18] Schultz RR, Stevenson RL. Extraction of high-resolution frames

form video sequences. IEEE Transactions on Image Processing

1996;5:996–1011.

[19] Patti A, Sezan MI, Tekalp AM. Superresolution video recon-

struction with arbitrary sampling lattices and nonzero aperture

time. IEEE Transactions on Image Processing 1997;6:1064–76.

[20] Shin JH, Jung JH, Paik JK. Regularized iterative image

interpolation and its application to spatially scalable coding.

IEEE Transactions on Consumer Electronics 1998;44:1042–7.

[21] Shin JH, Jung JH, Paik JK. Spatial interpolation of image

sequences using truncated projections onto convex sets. IEICE

Transactions on Fundamentals of Electronics, Communications,

Computer Sciences, June 1999.

[22] Paik JK, Katsaggelos AK. Image restoration using a modified

Hopfield network. IEEE Transactions on Image Processing

1992;1(1):49–63.

[23] Kim TK, Paik JK. Fast image restoration for reducing block

artifacts based on adaptive constrained optimization. Journal of

Visual Communication and Image Representation 1998;9(3):234–42.

[24] Kim TK, Paik JK, Won CS, Choe YS, Jeong J, Nam JY.

Blocking effect reduction of compressed images using classifica-

tion-based constrained optimization. Signal Processing: Image

Communication 2000;5:869–77.

[25] Paik JK, Park YC. An edge detection approach to digital image

stabilization based on tri-state adaptive linear neurons. IEEE

Trans. Consumer Electronics 1991;37(3):521–30.

[26] Paik JK, Park YC, Kim DW. An adaptive motion decision system

for digital image stabilizer based on edge pattern matching. IEEE

Transactions on Consumer Electronics 1992;38(3):607–16.

[27] Tekalp AM. Digital video processing. Englewood Cliffs, NJ:

Prentice-Hall, 1995.

[28] Andrews HC, Hunt BR. Digital image restoration. Englewood

Cliffs, NJ: Prentice-Hall, 1977.

[29] Shin JH, Yoon JS, Paik JK, Abidi MA, Fast superreolution for

image sequence using motion adaptive relaxation parameters.

Proceedings of the 1999 International Conference Image Proces-

sing, Kobe, Japan, vol. 3, October 1999. p. 676–80.

[30] Katsaggelos AK, Biemond J, Schafer RW, Mersereau RM. A

regularized iterative image restoration algorithm. IEEE Transac-

tions on Signal Processing 1991;39(4):914–29.

[31] Schowengerdt RA. Remote sensing: models and methods for

image processing, 2nd ed. New York: Academic Press, 1997.

[32] Lim JS. Two-dimensional signal and image processing. Engle-

wood Cliffs, NJ: Prentice-Hall, 1990.

[33] Unser M. Fast b-spline transforms for continuous image

representation and interpolation. IEEE Transaction on Pattern

Analysis and Machine Intelligence 1991;13:277–85.

[34] Parker JA, Kenyon RV, Troxel DE. Comparison of interpolating

methods for image resampling. IEEE Transactions on Medical

Imaging 1983;2:31–9.

[35] Hong KP, Paik JK, Kim HJ, Lee CH. An edge-preserving image

interpolation system for a digital camcoder. IEEE Transactions

on Consumer Electronics 1996;42:279–84.

[36] Hong MC, Kang MG, Katsaggelos AK, An iterative weighted

regularized algorithm for improving the resolution of

video sequences. Proceedings of the 1997 International Con-

ference Image Processing, Washington, DC, vol. 2, October 1997.

p. 474–7.

[37] Tom BC, Katsaggelos AK, An iterative algorithm for improving

the resolution of video sequences. Proceedings of the SPIE Visual

Communication, Image Processing, Orlando, FL, March 1996.

p. 1430–8.

[38] Schultz RR, Stevenson RL. Extraction of high-resolution frames

form video sequences. IEEE Transactions on Image Processing

1996;5:996–1011.

[39] Hardie RC, Barnard KJ, Armstrong EE. Joint map registration

and high-resolution image estimation using a sequence of under-

sampled images. IEEE Transactions on Image Processing

1997;6:1621–33.

[40] Hardie RC, Barnard KJ, Bognar JG, Armstrong EE, Watson EA.

High-resolution image reconstruction from a sequence of rotate

and translated frames and its application to an infrared imaging

system. Optical Engineering 1998;37:247–60.

[41] Patti AJ, Sezan MI, Tekalp AM, High-resolution standards

conversion of low resolution video. Proceedings of the 1995

International Conference on Acoustics, Speech, Signal Processing,

Detroit, MI, 1995. p. 2197–200.

S. Kim et al. / Real-Time Imaging 9 (2003) 61–7272