LOWER ATTENTIVE REGION DETECTION FOR VIRTUAL CONTENT INSERTION IN BROADCAST VIDEO

ICME 2008Huiying Liu, Shuqiang Jiang, Qingming Huang, Changsheng Xu

Outline

Introduction LAR detection framework LAR detection for sports video VCI methods Experiments Conclusion

Introduction

Introduction

Virtual Content Insertion (VCI) is an emerging application of video analysis and has been studied for several years.

The task of VCI is to make the inserted content attractive to the viewers and meanwhile not intrusive.

Compared with time point, spatial position is even more important as improper placement will make the insertion intrusive.

LAR detection

Framework In our work we perform LAR detection with

shot as the spatio-temporal context.

LAR detection

Attention analysis : static saliency map + motion saliency map

A region contains more perceptive information than a pixel or a block and can be obtained by image segmentation▪ Image segmentation DENSITY-BASED CLUSTERING [12]

[12] Q. Ye, W. Gao, W. Zeng, Color Image Segmentation Using Density-Based Clustering, ICME 2003.

LAR detection

Attention analysis Static saliency

[5] H. Liu, S. Jiang, Q. Huang, C. Xu, and W. Gao. “Region-Based Visual Attention Analysis with Its Application in Image Browsing on Small Displays”. ACM Multimedia,2007

feature contrast

area factor

spatial distance between the two regions

adjacency degree

central effect

LAR detection

Attention analysis Motion saliency [8]

▪ The MVS visualizes the MVF and provides us with a visual aid to understand.

[8] L-Y. Duan, M. Xu, Q. Tian, C-S. Xu, J. S. Jin, “A Unified Framework for Semantic Shot Classification in Sports Video” IEEE Multimedia,2005

Angle MagnitudeTexture

Hue SaturationValue

LAR detection

Information analysis : information map + entropy map The necessity of a region is evaluated by its

information and entropy in our work. An LAR must supplies less information and

contains less entropy at the same time. In this paper a simple and effective method

is adopted to calculate the information and entropy of each region and the result is used to evaluate the region’s necessity.

LAR detection

Information can be calculated as:

Entropy of the region can be calculated as:

The final LAR should be of less information and less entropy.

H :the normalized accumulative histogram of a shoth :the normalized histogram of a RegionA : the region’s area

Example of bottom-up LAR Detection

Fusing the 4 maps generates the frame attention map.

(a): The previous frame(b): Current frame(c): Motion field(d):Segmentation result(e): Static saliency map (f ): Motion intensity(g): Texture by DCT AC energy(h): HSV image of motion vector(i): Motion saliency map(j): Information map.(k): Entropy map(l): Final attentive map

LAR detection for sports video

Method Overview The shots here are classified into field view,

player view (coach , referee) and audience view.

Filed view and player view shots take up more than 90% of the video. So we perform LAR detection and VCI only on filed view and player view shots.

For player view shots LAR can be detected by using the bottom up method.

LAR detection for sports video

Field View LAR Detection Playfield detection, line and curve

detection, as well as object detection [2] can be combined to detect the LAR.

The GMM based method [9] is adopted to detect the playfield.

[2]C. Xu, K. W. Wan, S. H. Bui, Q. Tian, “Implanting Virtual Advertisement into Broadcast Soccer Video”, PCM, 2004[9] S. Jiang, Q. Ye, W. Gao, T. Huang. “A New Method to segment Playfield and Its Applications in Match Analysis in Sports Video”. ACM

Multimedia, 2004

Virtual content insertion

The methods for VCI include Static insertion▪ The content floats over the original video.

Dynamic insertion▪ Merge the content into the background.▪ Camera parameter must be reconstructed and

the content is adapted to the camera metric [10].

input Static insertion

dynamic insertion[10] X. Yu, X. Yan, T. T. P. Chi and L. F. Cheong, “Inserting 3D projected virtual content into broadcast tennis

video”, ACM Multimedia, 2006

Experiments

Test sequences (MPEG format) Broadcast TV play series (22 minutes) Sports1 soccer video and (45 minutes)

Sports2 tennis videos (26minutes)

For the field view shots, we insert the virtual content using dynamic method. The VCI method presented in before

estimates the camera motion and adapts the VC to the camera metric.

Experiments

Static insertion

dynamic insertion

Conclusion

In this paper, an LAR detection method is proposed using visual attention analysis and information theory.

In the future we will construct an integrated system of VCI and the temporal position for VCI.