DEPTH IMAGE-BASED TEMPORAL ERROR CONCEALMENT FOR3-D VIDEO TRANSMISSION

Yunqiang Liu, Jin Wang, and Huanhuan Zhang

IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, VOL. 20, NO. 4, APRIL 2010

Professor: Jar - Ferr YangPresenter: Jen - Hung Yeh

OUTLINE

Introduction Depth Image-Based Temporal Error Concealment

A. Error Concealment for Depth Stream B. Error Concealment for 2-D Video Stream

Homogeneous MB Boundary MB

Simulation Results Conclusion

INTRODUCTION(1/2)

3-D television (3DTV) system has recently received increasing attention and is believed to be the next logical development.

In DIBR-based 3-D video application, it requires transmitting 2-D video and its corresponding depth information.

In the video transmission over unreliable channels such as wireless or internet, the transmission errors, such as packet losses and bit errors, are inevitable.

INTRODUCTION(2/2) Therefore , there is a need for good error

concealment (EC) algorithms to compensate the large drop in received video quality.

It explores the correlation between the 2-D video stream and the depth stream and proposes a novel temporal error concealment approach by taking advantage of the depth information.

DEPTH IMAGE-BASED TEMPORAL ERROR CONCEALMENT

A. Error Concealment for Depth Stream B. Error Concealment for 2-D Video Stream

ERROR CONCEALMENT FOR DEPTH STREAM(1/2)

The MV from the corresponding 2-D frame can be taken as the recovered MV for the corrupted MB in the depth stream.

In case of the error occurring at the same location in the depth stream and the video stream, the DMVE algorithm is performed in the depth sequence to recover the lost MV.

ERROR CONCEALMENT FOR DEPTH STREAM(2/2)

The depth map does not contain any texture information; moreover, it cannot distinguish the different objects at the same distance with the camera, even if they have relative motion.

It is not appropriate to take directly the MV of depth map as the recovered MV for 2-D video.

ERROR CONCEALMENT FOR 2-D VIDEO STREAM

In the following steps, we will only consider the situation that the error occurs in the 2-D video.

Belong to two categories & The

difference between their average depth

values>T1

8-category histogram for the depth value

The lost MBs

YESNO

Boundary MBHomogeneous MB

<T2

HOMOGENEOUS MB

Find MV candidates of the MBs around the The MV of the

corresponding MB in depth mapFilter these MVs

candidate

The MV with minimumdiscard

YESNO

MViThe referenced depth block should coincide with that of the .

HOMOGENEOUS MB

(SAD represents sum of absolute differences) is calculated to measure the similarity between the two depth MBs

For 3-D video , we introduce a new matching criterion with consideration of the corresponding depth, which is defined as

BOUNDARY MB

It concludes four steps as follows: Step 1: MB Segmentation: Step 2: MV Candidates’ Initialization: Step 3: MV Selection: Step 4: Motion Compensation:

STEP 1: MB SEGMENTATION:

The lost MB

The pixels with the large depth value

Background Foreground

We take the foreground part as an example to describe the approach.

YESNO

STEP 2: MV CANDIDATES’ INITIALIZATION:

If the depth value of the grid is close to the average depth of the foreground area, both the MV of the grid in 2-D video and depth map are chosen as the MV candidate for the EC of foreground area.

And the collocated MV in the reference frame is also taken as the MV candidate.

Background

Foreground

The neighbors of the foreground area

Strong correlation with the lost foreground part

Similar with the recovery of homogeneous MB, the matching criterion considers both the neighbor information and the corresponding depth.

The matching criterion is defined as:

STEP 3: MV SELECTION:

measures the temporal correlation of the 2-D video , which is defined as follows:

measures the similarity between the current depth MB and its referenced MB , which is defined as:

STEP 3: MV SELECTION:

STEP 4: MOTION COMPENSATION:

The foreground area in the lost MB is recovered using the corresponding area in the reference frame according the recovered MV.

The background area is recovered with a similar process.

SIMULATION RESULTS(1/4)

depth map

2-D video

SIMULATION RESULTS(2/4)

Fig. 2. Comparison of PSNR for each frame with 5% packet loss.(a) Little Girl. (b) Mobile Phone. (c) Orbi.

SIMULATION RESULTS(3/4)Homogeneous MB

Boundary MB

SIMULATION RESULTS(4/4)

Homogeneous MB

Boundary MB

CONCLUSION

A depth image-based error concealment approach exploits the correlations between the 2-D video and its corresponding depth map to recover the lost blocks.

This strategy makes it preserve the temporal and spatial consistency on both 2-D video and depth information.