Personalized Abstraction of Broadcasted American Football Video by Highlight SelectionNoboru Babaguchi (Professor at Osaka Univ.)

Yoshihiko Kawai and Takehiro Ogura (NHK)Tadahiro Kitahashi (Professor at Kwansei Gakuin Univ.)IEEE Transactions on Multimedia, 2004

Outline

Introduction Related Work Method of detecting significant events in

video stream Method of generating video abstracts Experimental results Conclusion

Introduction

Video abstract Creating shorter video clips or video posters from an origin

al video stream Two schemes of video abstraction

Temporally compressing the amount of the video data Smith et al., Lienhart et al., He et al., Oh et al., Babaguchi

Provide image keyframe layouts representing the whole video contents Yeung and Yeo, Uchihashi et al., Chang et al., Toklu et al.

Introduction

This method of abstracting sports video Specifically broadcasted TV programs of America

n football Take personalization into consideration Belong to first scheme of video abstraction Abstraction based on highlights that are closely rel

ated to semantical video contents Detecting significant events like score events

Introduction

How to detect events Image analysis is very difficult This method’s solution is to make use of external metadata,

called gamestats Linking video segments with descriptions of the gamestats

Personalization Extensively attempted in a variety of application fields Emphasize it because the significance of scenes vary acco

rding to preferences and interests Provide a profile to collect personal preferences

Related work – time compression Smith et al.

Extracted significant information from video such as keywords, specific objects, camera motions and scene breaks with integrating language, audio, and image analyzes

Lienhart et al. Assemble and edit scenes of significant events in action m

ovies, focusing the on actor/actress’s closeup, text, and sound of gunfire and explosion

These two method are based on surface features of the video rather than on its semantical contents.

Related work – time compression Oh et al.

Abstracting video using user selected interesting scenes Automatically uncover the remaining interesting scenes in t

he video by choosing some interesting scenes Babaguchi

Video abstraction based on its semantical content in the sports domain

To select highlights of a game, an impact factor for a significant event in two-team sports was proposed

He et al. Create summaries for online audio-video presentations Use pitch and pause in audio signals, slide transition points

in the presentation, and users’ access patterns

Related work – spatial expansion Goal

Visualize the whole contents of the video Yeung and Yeo

Automatically create a set of video posters (keyframe layouts) by the dominance value of each shot

Uchihashi et al. Making video posters whose size can be changed accordin

g to the importance measure Chang et al.

Make shot-level summaries of time ordered shot sequence or hierarchical keyframe clusters, as well as program-level summaries

Detection of significant events Detect significant events in the original video stream

according to the description in the gamestats

Identification of event frames

An event occurs in the shot including the event frame

Attempt to recognize text expressing the game time in the overlay, and then to identify an event frame

To identify the event frame, an overlay model is employed

Detection of event shots

A shot is defined as consecutive image frames at a single camera view

Classify the event shots into four types live-play, replay, pre-play, and post-play shot

Generation of personalized video abstract Generating video abstracts from the detected

significant events Select highlights of the game from all the events,

considering profile descriptions The generating rules for the video abstract:

Profile

A video abstract has to be personalized because significance of events could change individually

Provide a profile to collect personal preferences and interests, the items are: Favorite teams Favorite players Events to want to see Specifications

Range of the video stream to be abstracted Length of the abstract

Significance degree of events

The highlights of the game depend on the significance of each event, and significance can be estimated in terms of event rank, event occurrence time, and the profile

Event rank State change event (SCE): a score event can change the current state into a

different state Rank 1: SCE’s. Rank 2: not SCE’s, but exceptional score events Rank 3: events closely related to score events Rank 4: all other event that are not Rank 1 to 3 Rank based significance degree of event Ir:

where ri denotes the rank of the ith event Ei, αis a coefficient to consider how large the difference of the rank affects the significance

Significance degree of events Event occurrence time

The score events occurring at the latter or final stage of the game largely affect the result that should have great significance

Occurrence time based significance degree of event It:

where N is the number of all events, β is a coefficient to consider how large the occurrence time affects the significance

Profile Comparing the descriptions of the profile and the occurring event Profile based significance degree of event Ip:

where l denotes the number of descriptions that don’t coincide with each other, γ is a coefficient to consider how large the profile affects the significance

Significance degree of an event I:

Selection of highlights

To determine highlights, we concentrate on both priority order of shots and significance degree of events

Priority order of each shot segment Motion live shot Still live shot Motion replay shot Still replay shot Motion pre-play shot Still pre-play shot Motion post-play shot Still post-play shot

Selection of highlights

Experimental results

Experimental results

Experimental results

Two measures to evaluate the quality of the generated abstract

where N denote the number of highlights included in abstracts

Experimental results – effect of personalization Inclusion ratio: the ratio of the length of shots which are

concerned with the specified team to the total length

Experimental results – effect of personalization

4-symbol string in the cells of table represents each condition of the pre-play, live, replay, and post-play shots for the event

Experimental results – effect of personalization

Experimental results – effect of personalization

Conclusion

Based on the detected significant events by recognizing the textual overlays

Link the video contents with useful external metadata by using the gamestats

Three sorts of significance degrees play a central role in highlight selection

Remaining problems The method is for different two-team sports A tailoring mechanism for shots, adjusting for the total abstr

act Seek for a sophisticated way of refining the profile