Methods of Video Object Segmentation in Compressed Domain

Cheng Quan Jia

Presentation Outline

• Features for Segmentation in Compressed Domain

• Using Motion Vectors in Segmentation

• Confidence Measure• Conclusion• Q & A

Features for Segmentation in Compressed Domain

An introduction to Compressed Domain

Compressed Domain: Definition

• Compressed Domain refers to video compression techniques that expliots Spatial and Temporal Redundancy using – DCT & Quantization– Motion Compensation

• Examples include MPEG-1/-2/-4, H.261 and H.263

Compressed Domain: Definition

• Opreations in the Compressed Domain involves processing of

– DCT coefficients (from I-macroblocks)– Motion Vectors (from P-/B-macroblocks)

Compressed Domain: Parsing

• Unlike pixel domain, operations in the compressed domain do not require the input bitstream to be decoded

• Instead, they are Parsed

Compressed Domain: Parsing

Features for Segmentation

• After Parsing, we have– DCT coefficients (from I-macroblocks)– Motion Vectors (from P-/B-macroblocks)

• Which coresspond to – Frequencies of texture change– Motion of the macroblock

Using Motion Vectors in Segmentation

Acquiring Dense Motion Field

• Many video object segmentation methods attempt to acquire a dense smooth motion field in order to create object masks

• For this end spatial interpolation and motion accumulation are employed

Motion Accumulation

Motion Accumulation

• Due to the different magnitude and signs of motion vectors, the obtained MVs are normalized, e.g. MVs in B-macroblocks would have their signs reversed

• Filtering is applied to remove non-uniform MV and smooth the motion field

Motion Accumulation

• Chen and Bajic [chen2009] employs MV Integration block-wise and pixel-wise to enhance the Motion Field

Motion Accumulation

Chen and Bajic [chen2009] Babu et al. [babu2004]

Porikli et al.’s Investigation

• The Compression Domain segmentation system published by Porikli et al. [porikli2010] experimented the effect of DCT coefficients and MV on segmentation performance– The DC parameters(for Y, U, V channels) of the I-

frame– Low vertical and horizontal frequency AC values– A spatial energy term– Aggregated motion flow of the corresponding

macroblock

Porikli et al.’s Investigation

• They create a Frequency-temporal data structure for each macroblock with the features and perform volume segmentation

• Their results show that using DCT terms in FT segmentation and using MV in the hierarchical clustering, on average, gives better results.

Porikli et al.’s Investigation

• The Block Matching Process in encoding stage looks for only the best match for a macroblock rather than object motion

Porikli et al.’s Investigation

Confidence Measure of Motion Vectors

• Coimbra and Davies [coimbras2005] try to approximate Lucas–Kanade optical flow in MPEG-2 Compressed Domain

Approximating Optical Flow

• They argue that AC[1] and AC[8] in an I-macroblock can be used as confidence measure

• The confidence update step will have a 8×8 macroblock referencing a 16×16 image block in the I-frame, and the confidence of the motion vector of the macroblock is the weighted average of confidence in the 16×16 window

Confidence Measure

Confidence Measure

Original image MPEG-2 smooth motion field afterconfidence threshold

Conclusion

• Due to block matching process, motion vectors in P-/B- frames do not necessary relate to object motion

• To ensure a motion vector is correlated to object motion, some sort of confidence measure is required

• [coimbras2005] demonstrated that edge strength can be an effective measure

Conclusion

• Problems not discussed here – Camera motion– Changes in illumination– Occlusions

Conclusion

References1. R. V. Babu, K. R. Ramakrishnan, and S. H. Srinivasan.

Video Object Segmentation: A Compressed Domain Approach. IEEE Transactions on Circuits and Systems for Video Technology, 14(4):462–473, April 2004.

2. Y.-M. Chen and I. V. Bajic. Compressed-Domain Moving Region Segmentation with Pixel Precision using Motion Integration. In IEEE Pacific Rim Conference on Computers and Signal Processing, 2009, pages 442 – 447, August 2009.

3. M. T. Coimbra and M. Davies. Approximating Optical Flow Within the MPEG-2 Compressed Domain. IEEE Transactions on Circuits and Systems for Video Technology, 15(1):103–107, January 2005.

4. F. Porikli, F. Bashir, and H. Sun. Compressed Domain Video Object Segmentation. IEEE Transactions on Circuits and Systems for Video Technology, 20(1):2–14, January 2010.

Q & A SECTION

Comments and Suggestions