Trajectory Analysis of Broadcast Soccer Videos Computer Science and Engineering Department Indian Institute of Technology, Kharagpur by Prof. Jayanta Mukherjee

Trajectory Analysis of Broadcast Soccer Videos

Computer Science and Engineering DepartmentComputer Science and Engineering Department

Indian Institute of Technology, KharagpurIndian Institute of Technology, Kharagpur

by

Prof. Jayanta Mukherjee

[email protected]

Collaborators

•V. Pallavi --- research scholar. •Prof. A.K. Majumdar, CSE •Prof. Shamik Sural, SIT

OUTLINE• Motivation and Objective

• State Based Video Model

• Extraction of Features

• Trajectory Detection

• States and Event Detection

MotivationIncreasing availability of soccer videos

Soccer videos appeal to a large audience

Processing of soccer videos to deliver it over narrow band networks

Relevance of soccer videos drops significantly after a short period of time

Therefore soccer video analysis needs to be made automatic and the results must be semantically meaningful

State based Video Model

Video data model : representation of information contained in the unstructured video in order to support users’ queries.

State based model: states of soccer video objects and their transitions (due to some event).

State Chart Diagram for Ball Possession

Immediate Goal

Our objective is to identify these states and their transitions by analyzing the unstructured video.

Cinematic Features

– Shot Transitions

– Shot Types

– Shot Durations

Object Based Features

– Players

– Ball

– Billboards

– Field Descriptors

Features UsedFeatures Used

Shots can be classified into:

• Long shot

– Captures a global view of the field

• Medium shot

– Shows close up view of one or more players in a specific part of the field

• Close shot

– Shows an above-waist view of a single player

Shot classification

Typical long views in soccer videos

Grass covering entire frame Grass covering partial frame

Shot Classification (contd..)Shot Classification (contd..)

Soccer Video Sequence

If dominant color is green

Dominant color ratio >0.75 and

<=1.0

Long Shot Medium Shot


<=0.75

Close Shot


<=0.5

Shot Classification Results

20147211078Close Shot

73841837Medium Shot

14424325830Long Shot

Unclassified Shot

(No of frames)

Close Shot

(No of frames)

Medium Shot

(No of frames)

Long Shot

(No of frames)

Predicted ClassTrue Class

87.6387.63Close ShotClose Shot

83.7683.76Medium Medium ShotShot

96.6896.68Long ShotLong Shot

% of True % of True ClassificationClassification

Shot TypeShot Type

Shot Classification Results

Shot detection improved by shot classification

Object Based FeaturesFeature extraction for grass pixels

Each frame is processed in YIQ color space.

It is found experimentally that grass pixels have ‘I’ values ranging between 25 and 55 while ‘Q’ values range

between 0 and 12.

0 50 100 150 200 2500

50

100

150

200

250

Q v

alu

es

I values

Grass Values

Playfield region detected

Grass pixels detected for a long view frame

Object Based Features (contd..)

Playfield Line Detection

A playfield line separates playfield from the non playfield background which are usually the billboards (also called advertisement boards).

Hough transform is used to detect the playfield line.


Midfield line is the line that divides the playfield in half along its width. Hough transform is applied to detect the midfield line.

Midfield Line Detection

Ball DetectionObject Based Features (contd..)

Challenges :

Features of the ball (color, size, shape) vary with time

Relative size of the ball is very small

Ball may not be an ideal circle because of fast motion and illumination conditions

Objects in the field or in the crowd may look similar to a ball

Field appearance changes from place to place and time to time

No definite property to uniquely identify ball in a frame

Detecting Ball Candidates in Long Shots

• Obtain ball candidates by detecting circular regions by using circular Hough Transform

• Filter the non ball candidates by :

– Removing candidates from channel’s logo

– Removing candidates from gallery region

– Removing candidates from midfield line

– Filtering out the candidates moving against the camera


Ball candidates before and after filtering

Ball candidates before filtering Ball candidates after filtering


Detecting Players in Long Shots

Challenges

• Features of the players (color, texture, size, motion) are neither static nor uniform

• Players appear very small in size

• Size of players changes with their position and zooming of cameras

• Color and texture of the jersey and shorts vary from team to team

• Players in the field do not have constant motion


• Obtain player pixels by removing non player pixels :

– Removing grass pixels

– Removing the broadcasting channel’s logo

– Removing the extra field region (billboards and gallery)

– Removing pixels from the midfield line

• Segment the image containing player pixels to isolated player regions by :

– Region growing algorithm

– Center of the bounding rectangle of each region is said to be the location of the player

Detecting Player Regions


A Long Shot View


Player pixels detected


Players detected in long shot views


Team Identification in Soccer Videos

Players in a soccer videos are classified using a supervisory classification method.

Mean I and Q values of the player regions are obtained by randomly selecting a few frames

The minimum and maximum I and Q values are set as the range for classifying player regions

Feature Detection (Contd.)

Team Classification in Soccer Videos

Experiments were performed on two different matches:

Real Madrid and Manchester United (UEFA Champions League 2003)

Chelsea and Liverpool (UEFA Champions League 2007)

Feature detection (contd.)

Team Classification Results

72

(11.51)

6520Team B

50

(25)

3

(17.86)

Unclassified

173

(16.29)

0725Team A

UnclassifiedNo of players (%)

Team BNo of players (%)

Team ANo of players (%)


Real Madrid and Manchester United

Chelsea and Liverpool

72

(9.94)

6520Team B

503

(37.5)

Unclassified

173

(19.27)

0725Team A

UnclassifiedNo of players (%)

Team BNo of players (%)

Team ANo of players (%)


Team Classification Results (contd..)

Camera Related FeatureObject Based Features (contd..)

Camera Direction Estimation :

1. Optical Flow velocities and their directions are computed using Horn and Shunck’s method.

2. Based on the sign of the horizontal component of the majority pixels in a frame, the direction of movement (left or right) of the camera is estimated.

Camera Direction Estimation (contd..)

Optical flow velocities for the camera moving towards right

Tracking of Broadcast Video Objects

Challenges

Camera parameters are unknown

Cameras are not fixed

Cameras are zoomed and rotated

Broadcast video is an edited video

Construction of a Directed Weighted Graph

Objects in a frame form nodes.

Between two correlated objects in two different framesan arc (edge) is formed.

The measure of correlation or similarity provide the weight.

Temporal direction provides the direction of the edge.

Object Trajectory Detection

Given a source node, longest path of the graph obtained by dynamic programming gives the path of the object.

Tracking of Broadcast Video Objects (contd..)

Ball detection results for long shots

10010027272741400-41940

91.8598.03609597650Liang et al * sequence 1

95.8398.26689677719Liang et al * sequence 2

94.7494.7419181937020-37400

10010020202040500-40900

92.7396.2353515535900-37000

93.3393.3330283034800-35400

91.395.4522212330300-30760

9094.7319182023800-24200

PrecisionRecallBall present in

(number of frames)

Ball identified in

(number of frames)

Total

(number of frames)

Frame Range

Average Recall is 96.75 % and Average Precision is 94.42 % * Liang D., Liu Y., Huang Q. \& Gao W., A Scheme for Ball Detection and Tracking in Broadcast Soccer Video, Pacific Rim Conference on Multimedia, 2005, 1, LNCS 3767, 864-875.


Results for ball detection in long shots (contd..)

Given a source node (player in the first frame), longest path of the graph obtained by dynamic programming gives the path of the player in the whole sequence.

Tracking a Single Player

Player being tracked


Tracking Multiple Players

Longest path from each node (represented by players in the first frame) of the graph obtained by dynamic programming gives the trajectories of the players for the sequence of frames.

Limitations :

Occlusion between players

Players in contact

Similarity between players belonging to same team


Resolving Conflicting Player Trajectories

If more than one player has more than two common nodes in its trajectory then only one amongst them is true.

The path having maximum weight is said to be the true trajectory

Nodes constituting the paths of correctly detected players are removed and a graph is again constructed

Mistracked players are again tracked

Tracking Multiple Players (contd..)

Multiple Player Detection Results

99.3799.48158815781586317Soccer 6

95.2810053050550568Soccer 5

83.58100670560560100Soccer 4

10085.7136036042060Soccer 3

81.81100220018001800200Soccer 2

81.03100311025202520180Soccer 1

Precision (%)

Recall (%)

SOP detected SOTP detectedSOP presentNo of FramesVideo file

Average Recall is 97.53 % and Average Precision is 90.18%


Multi - Player Tracking Results

96.59153212901586317Soccer 6

91.4946235650568Soccer 5

100560540560100Soccer 4

90.4838030042060Soccer 3

100180012001800200Soccer 2

85.71216018002520180Soccer 1

Accuracy (%)

SOTP tracked by tracking

and retracking algorithm

SOTP tracked by tracking algorithm

SOP presentNo of FramesVideo file

Average Accuracy is 94.05 % .


Occlusion Results

1002020Soccer 6

801620Soccer 5

1004444Soccer 3

55.552036Soccer 1

Accuracy

(%)

No of cases that could be solved

No of occlusion and contact cases

Video file

Average Accuracy is 83.89 % .


Multi - Player Tracking Results

Multi - Player Tracking with Occlusion Results

Multi - Player Tracking with Occlusion Results (contd..)



Tracking the Mistracked Player (contd..)




Detection of States and Events

The features extracted and the trajectories detected are used to detect states and events based on the proposed state based video model.

States identified

- Ball possession states

Events detected

- Ball passing events

State Chart Diagram for Ball Possession

Detection of States and Events (contd..)

State Detection

Ball possession states are obtained based on

Spatial proximity analysis:

Distance between nearest player and second nearest player to the ball

Spatial arrangements between the players and the ball

Ball Possession State Detection

Ball in possession of player 1’s team


Ball Possession State Detection


Ball in a fight state

Ball Possession Results

8

(1.92)

39216

(3.85)

Team B

5412

(17.14)

4

(5.71)

Fight

8

(3.17)

2

(0.93)

206Team A

FightNo of frames

(% of misclassified

frames)

Team BNo of frames

(% of misclassified

frames)

Team ANo of frames

(% of misclassified

frames)


Edit Distance as performance measure for ball possession states

If the actual state sequence for a sequence of frames is:

AAAAFFFFFFFFFFBBBB

And if the state sequence obtained by the proposed algorithm is:

AAAAFFFFFFFFBBBBBB

Both the sequences are represented as strings S1 and S2.

Edit distance D(S1, S2 ) is defined as the minimum number of point mutations required to change S1 to S2 where a point mutation is one of:

replacing an alphabet

inserting an alphabet

deleting an alphabet

Edit distance for the above sequence is 2. While normalized edit distance is:

D(S1, S2 )/| S1 |

|S|)S,S(D 121

Shot wise ball possession results

Event Detection

The event detected in this work is the ball passing event. It can be:

Forward pass

Reverse pass

Event Detection (contd..)

The ball passing event cannot be detected

from state transition graphs because: Ball is usually passed between players of the same team State transition graphs show the change in ball possession states from Team A-Team B, Team B - Team A, Team B – Fight , Fight – Team B, Team A – Fight or Fight – Team A

Schematic diagram for ball passing events

Ball is said to be passed in a sequence of frames, if:

Nearest player in the initial frames of the sequence is the second nearest player to the ball in the subsequent frames

Nearest and the second nearest players to the ball belong to the same team

Example of a ball passing event:

Example of a ball passing event (contd..)



Classifying ball passing events

Forward pass:

Direction of camera motion is towards the goal post of the team opposite to that of the nearest player

Reverse pass:

Direction of camera motion is towards the goal post of the team of the nearest player

Results for ball passing events

Average Recall = 100% and Precision = 60%

Classification of ball passing events:

5-Reverse

13Forward

Reverse

(no of passes)

Forward

(no of passes)

False Ball PassesTrue Ball Passes

Graphs for ball possession and ball passing

Graphs illustrating ball possession states and ball passing events for Sequence 7

Graphs for ball possession and ball passing

Graphs illustrating ball possession states and ball passing events for Sequence 10

Documents

Trajectory Analysis of Broadcast Soccer Videos Computer Science and Engineering Department Indian Institute of Technology, Kharagpur by Prof. Jayanta Mukherjee