Occlusion Adaptive Frame Rate Up-Conversion

Burak Özkalayc�, A. Ayd�n Alatan Department of Electrical and Electronics Engineering

METU Balgat 06531 Ankara Turkey

e-mail: bozkalayci@gmail.com, alatan@eee.metu.edu.tr

Ahmet Ba�tu� Vestek R&D

ITU Ayaza�a ARI-2-B Blok Maslak 34469 Istanbul Turkey

e-mail: ahmet.bastug@vestel.com.tr

Abstract—This paper presents a novel occlusion adaptive frame rate up-conversion method. The proposed method consists of two main steps; true motion estimation and directionally adaptive motion compensated interpolation. In order to reliably estimate the motion vectors of occlusion regions without increasing the number of frames, forward and backward directed motion vectors are merged recursively. Following the motion estimation step, a motion compensated interpolation is done by determining the temporal interpolation source directions under an OBMC framework. The experimental results show that the proposed approach improves the frame rate up-conversion performance and reduce the occlusion related halo artifacts compared to the conventional approaches.

I. INTRODUCTION The motion compensated Frame Rate Up-Conversion

(FRUC) approaches estimate the motion vector field of the frame and makes a motion compensated interpolation between the original frames of the video. The correctness of the estimated motion vector field directly affects the quality of the temporal interpolation of the video hence the temporal flow of the output video. Although true motion estimation techniques are developed in the literature for FRUC applications, the motion estimation errors at occlusion regions brings the annoying hot air like artifact, called as halo artifact, around the object boundaries. Various halo artifact reducing FRUC approaches are introduced in the literature and they can be classified into two main approaches. The first approach detects the occlusion regions and corrects the erroneous motion vectors explicitly [1] and the second approach implicitly tries to correctly estimate the motion vectors of the occlusion regions at the motion estimation step [2]. After acquiring the motion vector field estimate, the motion compensated interpolation step is also done by taking care of the occlusion regions.

In the forthcoming sections a novel occlusion handling motion compensated FRUC algorithm will be introduced. The motion estimation and the motion compensated interpolation steps will be mentioned in sections 2 and 3 respectively. The experimental results will be given in section 4 and section 5 will conclude the paper.

II. TRUE MOTION ESTIMATION In order to correctly interpolate the video in temporal

domain, the motion compensated interpolation methods need reliable true motion vectors of the objects in the frame. In

order to constraint the true motion estimation problem some general assumptions are used in the literature. In general, widely used photoconsistency assumption is regularized with the assumptions that the neighboring regions of the objects move in similar directions and the objects have slowly varying inertia. By these assumptions the true motion estimation algorithms can provide spatially and temporally smooth motion vector fields.

Block based and local algorithms like 3 Dimensional Recursive Search (3DRS) [3] are preferred widely for FRUC applications. The 3DRS approach uses a limited number of motion vector candidates which are selected from spatio-temporal neighborhood and it converges to true motion vectors by adding small update perturbations to some of the candidates. The 3DRS algorithm assigns a motion vector to every block in a proper scan order and the assigned motion vector satisfies the minimum SAD matching cost, equation (1), among the recursively updated candidate set. The � variable in (1) is the temporal interpolation reference between 0 and 1.

�∈

+ −+−−BX

tt VXIVXI ))1(()( 1 αα (1)

However the SAD matching cost is not a reliable estimation metric at occlusion regions since cover and uncover regions are not visible at both time instances, t and t+1. Hence the motion vectors acquired for cover and uncover regions according to (1) are error prone as illustrated in Fig. 1-a. Increasing the number of frames for motion estimation and special retiming operations are proposed in the literature for reliably estimating the motion vectors of occlusion regions [2]. Similar to retiming operations, the proposed method estimates the motion vectors of the occlusion region by merging the forward and backward motion vectors recursively but without increasing the number of frame buffers. The forward and backward motion vectors are obtained by two separate 3DRS routines for � equals to 0 and 1 respectively. The forward and backward motion vectors of the visited block are updated consecutively.

The forward motion vector estimating 3DRS routine can provide the uncover regions reliably but it is error prone at cover regions and vice versa for the backward motion estimating 3DRS routine. The complementary nature of the forward and backward routines are illustrated in Fig. 1-b and 1-c respectively. The proposed method merges the forward and backward motion vector fields in a way that selects the

This work is supported by Vestek R&D

2011 IEEE International Conference on Consumer Electronics - Berlin (ICCE-Berlin)

978-1-4577-0234-1/11/$26.00 ©2011 IEEE 165

Figure 1. Color coded motion estimation results; (a) conventional, (b) forward, (c) backward, and (d) merged

motion vectors from forward field for uncover regions and the motion vectors from backward field for cover regions. The motion vector update of the merged field is done as selecting one of the updated vectors between the forward and backward directions. In order to keep the practical recursive property of 3DRS algorithm the block updates of the merged field are done just after block updates of the forward and backward fields.

The SAD matching costs of the forward and backward fields are compared in consideration of a local vector field smoothness constraint for updating the merged field as given in (2). The local smoothness constraint is calculated in the 8-neighborhood of the block in the merged vector field. The recursive merging result of the shown forward and backward fields are given in Fig. 1-d.

{ } �∈∈

−+=NV

nVVVmrgn

bwdfwd

VVVSADV λ)(minarg,

(2)

III. DIRECTIONAL OBMC The motion compensated interpolation techniques used for

FRUC applications need to synthesize subjectively satisfactory and artifact free images. The main sources of the interpolation artifacts are erroneous motion vector estimates, occlusion regions and block precision motion vector field acquisition. The erroneous motion vectors and occlusion regions may cause disturbing regionally uncorrelated texture synthesis and block based motion estimation may cause blocking artifacts.

In order to avoid blocking artifacts the Overlapped Block Motion Compensation approaches [4] are proposed in the literature. In OBMC, the neighboring motion vectors also contribute to interpolation of a block and it provides smooth transitions across the block boundaries. Occlusion related

TABLE I. SYNTHESIS EQUATIONS

Synthesis equations according to direction flags 1== bwdfwd ff ))1(()()1()( 1 mrgtmrgtt VXIVXIXI ααααα −++−−= ++

1,0 == bwdfwd ff )()( mrgtt VXIXI αα −=+

0,1 == bwdfwd ff ))1(()( 1 mrgtt VXIXI αα −+= ++

0== bwdfwd ff )()()1()( 1 XIXIXI ttt ++ +−= ααα

artifacts are also reduced by median filter like ordered statistics based interpolation filters [2] or by explicit motion vector correcting processes [1].

In the proposed motion compensation approach the erroneous motion vectors and occlusion regions are handled by checking the motion vector flow continuity direction. The flow continuity is checked by motion vector similarity thresholding in the forward and backward directions of the motion vector as given in (3).

TVXVXVf

TVXVXVf

mrgmrgmrgbwd

mrgmrgmrgfwd

<−−=

<−+−=

)()(

))1(()(

μα

αμ (3)

According to thresholding, forward and backward interpolation flag assignments are determined for each motion vector. While the forward interpolation flags get false value at cover regions, the backward interpolation flags get false value at uncover regions. The motion vectors whose forward and backward flags both get false value are expected to be erroneous estimates since their flow are not continuous in both directions.

In the guidance of the interpolation flag values the proposed synthesis equations are given in Table 1. The first case is compensated with the traditional bidirectional synthesis which uses the nearest two frames. The second and the third cases are compensated with the proposed unidirectional synthesis which uses only the previous or the next frame for cover and uncover regions respectively. And the last case is compensated with the bidirectional synthesis by replacing the possible erroneous motion vector with the null vector.

In addition to block-wise motion estimation the proposed synthesis equation brings also directional source discontinuity

Figure 2. Motion compensation weights of blocks in OBMC.

166

Figure 3. Color coded source frame direction for synthesis

related blocking artifacts. In order to avoid blocking artifacts, the OBMC approach is utilized. The overlapping parts of the OBMC method are extended to cover the 8-neighborhood of the block of interest. The weighted contributions of a motion vector to its own and neighboring blocks are given in Fig. 2. In order to minimize the contribution of nullified erroneous motion vectors, the weights are kept at minimum. Since the sum of all weights for a block depends on the forward and backward flag values of the contributing motion vectors, a weight accumulation buffer is used in order to obtain the intensity values of the interpolated frame after a weight normalization step.

Utilization of the OBMC method with the proposed direction adaptive synthesis provides occlusion adaptive smoothly varying transitions around the occlusion regions. The source frame selection of the proposed synthesis equations are illustrated in Fig. 3. The red and green regions corresponds to 2nd and 3rd cases of Table 1 which are in accordance with the uncover and cover regions.

IV. EXPERIMENTAL RESULTS In order to evaluate the performance of the proposed

FRUC algorithm objectively and subjectively, it is compared with a conventional FRUC approach. The 3DRS algorithm for � equals to 0.5 and the OBMC based motion compensated interpolation method using only the first equation of Table 1 are utilized for motion estimation and frame interpolation steps of the conventional FRUC method respectively. All 3DRS routines are implemented in multi-resolution from16x16 to 2x2 block sizes.

In order to compare the interpolation performance, the videos captured at 60 Hz [5] are down-sampled to 30 Hz and then up-converted to 60 Hz by the conventional and the proposed FRUC approaches. The PSNR values of the interpolated frames are calculated with respect to original 60 Hz videos. The PSNR values of both approaches are given in Table 2.

TABLE II. MEAN PSNR VALUES

Conventional Proposed City 28.40 29.78

Crew 30.52 30.51

Harbour 30.46 31.20

Soccer 28.82 29.28

And for the subjective comparison of the conventional and proposed methods in handling the occlusion related halo artifacts, the interpolation results of the two methods are give in Fig. 4. The test video used in this experiment has foreground and background objects moving in highly different directions. While the halo artifacts around the foreground objects are visible on the interpolation of conventional approach, they are negligibly noticeable on the interpolation of proposed approach.

V. CONCLUSIONS A practically implementable FRUC approach handling the

occlusion regions and reducing halo artifacts without increasing the frame buffer is introduced. The proposed occlusion handling mechanism has two main novelties in motion estimation and compensation steps respectively. While the recursive merging of the forward and backward directional motion estimates provides more reliable motion vectors at occlusion regions, the motion flow continuity related directional OBMC approach reduces possible interpolation artifacts. The objective and subjective comparison of the proposed approach with a conventional approach demonstrates that the proposed method is effective and promising for FRUC applications.

REFERENCES [1] A.M. Huang and T. Nguyen, “Correlation-based motion vector processing with adaptive interpolation scheme for motion-compensated frame interpolation,” IEEE trans. Image Processing, vol 18, pp. 740-752, Apr 2009 [2] E.B.Bellers, et al., “Solving occlusion in Frame Rate Up-Conversion,” ICCE, pp. 1-2, Jan. 2007. [3] G. de Haan, et al., “True motion estimation with 3-D recursive search block matching,” IEEE trans. Circuits and Systems for Video Technology, vol. 3, pp. 368-379, 1993. [4] C. Auyeung, et al., “Overlapped Block Motion Compensation,” Proc. SPIE, vol. 1818, pp. 561-572, 1992. [5] ftp://ftp.tnt.uni-hannover.de/pub/svc/testsequences/, Jan. 2011.

Figure 4. Halo performances of conventional and proposed approaches from left to right and close-up details in the same order.

167