1

Structural Alignment-Based Temporal Concealment of Packet-Loss in Video

Ajit S. Bopardikar, Odd Inge Hillestad and Andrew Perkis

Centre for Quantifiable Quality of Service in Communication Systems

Trondheim, Norway.

2

Publications Based On This Work

• A. S. Bopardikar, O. I. Hillestad and A. Perkis, “Temporal error concealment algorithm based on structural alignment for packet video,” International Conference on Signal Processing and Communications (SPCOM), Bangalore, India, December 2004.

• A. S. Bopardikar, O. I. Hillestad and A. Perkis, ”Structural alignment-based temporal concealment of packet-loss in video” in review, International Conference on Acoustics Speech and Signal Processing, Philadelphia, PA, USA, May 2005.

3

This Presentation Is About….

• A novel temporal algorithm to conceal packet-loss related distortion in video.

• Algorithm based on the insight that the spatial region around the area in the previous frame that yields the best concealment will in general be highly correlated with the corresponding area around the region to be concealed in the present frame.

4

Overview

• Packetization overview

• Packet-loss related artifacts in block-based video compression schemes.

• Spatial and temporal concealment methods.

• BMA-type temporal concealment methods.

• Proposed method.

• Results and discussions.

5

Flowgraph for Video Stream Packetization

Video stream

RTP/UDP

IP packets

Internet/Simulation

Testnet

Smart mapping

Flows

File format

6

Example: MPEG-2 Bitstream

Transport stream layer

6 byte header

PES layer

Picture Header

Picture Layer (containing n slices)

Slice 1 Slice n

Slice Header

Slice Layer (containing m macroblocks)

MB 1 MB m

MacroblockHeader

Macroblock Layer (containing k blocks)

Block 1 Block k

Macroblock location,motion vector information

header

7

MPEG-2 Bitstream

• Each packet can be decoded independently irrespective of the errors in decoding previous packets.

• Loss of packet means loss of spatial and motion vector information for that part of the frame.

8

Packet-loss Related Artifacts in Block-Based Coding Schemes

9

Concealment Methods for Packet-Loss Related Errors in Video

• Spatial

• Temporal

10

Spatial Concealment Methods

• Bilinear interpolation based methods.

• Projection onto convex sets (POCS).

• Requires no knowledge of the previous frame(s).

• Simple and can cause blurring.

• Better results can be obtained using temporal methods.

11

Temporal Concealment Methods

• Makes use of the high degree of correlation that generally exists between consecutive frames.

• The idea is conceal artifacts using ’best-fitting’ parts from the previous frame which is assumed to be error-free.

• Examples: – Temporal Replacement (TR)

– Boundary Matching Algorithm (BMA).

12

Temporal Replacement

• Replaces lost and damaged macroblocks (MBs) in the present frame with MBs from the same spatial location in the previous, error-free frame.

• In presence of significant motion, TR can give rise to visible discontinuities at boundaries of replaced MBs.

13

Boundary Matching Algorithm

• Motion vector information typically lost with the packet.

• These motion vectors need to be estimated for effective concealment.

• BMA assembles a set of candidate motion vectors and then chooses a best fit from among the MBs in the previous frame associated with them.

• Best fit MB minimizes the sum of mean squared error (MSE) or the sum of absolute difference (SAD) between its boundaries and the boundaries adjacent to them from the top, bottom and left MBs around the area to be concealed.

Candidate

MB

14

Proposed Algorithm

• Based on the high degree of correlation that can exist between spatial region around the best-fit candidate MB in the previous frame and the corresponding region around the MB-area to be concealed in the present frame.

F-1 F0

n

n

n

n

n

n

n

n

M0T

M0B

M0C

M-1T

M-1C

M-1B

M0L

M-1L

15

Proposed Algorithm

• Insert candidate MB (M-1C) in the area to be concealed.

• Compute x and y Sobel gradient maps for the two composite blocks.

• Compute the overall magnitude gradient map.

• In the gradient map, compute the SAD between the corresponding red and green boundaries of the surrounding MBs.

• Choose the MB from the previous frame that minimizes the sum of the three SADs for concealment.

M-1Comp M0Comp

M-1T

M-1C

M-1B

M-1Comp(n-1,i)

M-1Comp(2n,i)

M0Comp(n-1,i)

M0Comp(2n,i)

M0T

M-1C

M0B

M0LM-1L

M0Comp(j,n-1)

M-1Comp(j,n-1)

16

Simulations: Particulars

• Full search adopted to search for the best fitting MB.

• The area in the previous frame for the search is 52x52 pixels centered on the spatial position of the MB to be concealed.

• All video clips in SIF (352x240 pixels, 30 frames per second) MPEG-2 coded at 1.5Mbps.

• Packet-loss simulated using NTT DoCOMo software. PLR = 0.001.

• Performance compared to a BMA-like algorithm.

17

Simulations: Results for Frame 346 of ’Table Tennis’ sequence.

Original PL (13.84 dB)

BMA (28.91 dB) Proposed (36.34 dB)

18

Simulation Results: PSNR

Sequence Fr. no. PL

(dB)

BMA

(dB)

Proposed

(dB)

Susie 67 17.53 38.68 39.91

120 11.17 29.35 31.07

208 10.55 28.29 39.42

Mobile 72 21.93 27.13 29.58

139 13.19 22.65 30.17

202 14.02 24.94 39.07

Flower Garden

46 8.96 19.35 26.64

67 8.92 19.99 28.75

208 14.32 27.30 30.47

Table-tennis

304 14.89 28.05 32.93

325 15.72 25.98 29.31

19

Conclusions

• Proposed algorithm better at aligning edges.

• Relies on matching the structure around the MB to be concealed and the candidate MB.

• Low computational complexity.