Robust Visual Golf Club Tracking

Vincent Lepetit, Nicolas Gehring, Pascal FuaCVlab - EPFL

2

Aim

• Completely automatic video based system.– No user intervention.– Use of a single video (PAL) camera.– No external expensive devices.– No specifically instrumented golf clubs or clothing.

• Usable in natural environments– Cluttered (fixed) background

1.5 sec

3

Club: thin, specular reflexion, moves very fast (up to 170km/h).

sec25

1+tt

4

• Club extraction• Tracking algorithm with

– local motion model– global motion model

5

Dealing with interlaced images• For each frame:

50 « half-images » per second

2 « half-images » from an interlaced image

6

Detection of moving objects

Threshold + Closing

Threshold + Closing

-

-AND

Mask of themoving objects in frame n

Frame n

Frame n-1

Frame n+1

7

Hypothesis generationDetection of adjacent parallel segments under the moving-object mask1. Edges detection

2. Segment detection (contour extraction, polygonal approximation)

3. Parallel segment detection and fusion

8

Some results of the parallel segments detection

Hypothesis generation

9

Hypothesis generation• The resulting segment is only a part of the shaft

Search for the shaft end-points

• Looking for the club head in the moving-object mask (as the last white point)

• Looking for the “hands” in the color image

10

• Results– In this example, two hypothesis:

– Others results:

Hypothesis generation

11

• Heuristics for removing some hypotheses– Given a 2D point somewhere between the golfer shoulders,we can remove some physically impossible hypotheses

Hypothesis generation

ImpossiblePossible

Shoulders Shoulders

• No accurate position for this point needed• Can be easily provided by the user

12

• Some wrong hypotheses can not be removed:

Why we still need a tracking algorithm ?

?

?

??

13

Tracking

• Many visual tracking techniques have been proposed in the computer vision literature.– Data Association approaches (MHT)– ConDensation

– Based on recursive motion models: Xt+1 = f(Xt)– Difficult to consider a specific motion such as a golf

swing.– Suffer from a lack of robustness for practical

applications when:• Frequent mis-detections• Large acceleration • Abrupt motion changes

14

New tracking algorithm

• Idea:– Take into account previous frames + next

frames– Consider the detections in these frames to

locally estimate the club shaft motion

15

New algorithmMLESAC applied to tracking:• Choose randomly 3 frames (in the previous and next frames)• Choose randomly one detection in these frames• Compute the shaft motion assuming a locally constant

acceleration• Estimate the shaft position in the previous and next frames

Several examples:

• Compute the support of the predicted motion i.e. the number offrames where there is a detection near the predicted position

• Repeat and keep the shaft motion with the maximum likelihood

Deals easily with mis-detections and false-alarmsRobust motion estimation

16

• Parametrisation of the shaft (double-pendulum model):

• Estimation– From the three randomly selected shafts si, sj, sk,– assuming a constant acceleration for all the parameters,

• we can predict the position, velocity and acceleration of the shaft

s 0 = [Shoulders 0, L 0, R 0, Ψ 0, ϕ 0] in the current frame:

• we can also predict the position in the other frames:

Motion estimation

s = [Shoulders, L, R, Ψ, ϕ] Lϕϕϕϕ

ψψψψ

R

������

�

�

������

�

�

−

−

−

=

2

)1(.1

2

)1(.1

2

)1(.1

kkk

jjj

iii

A

���

�

�

���

�

�

ΨΨΨ

=���

�

�

���

�

�

ΨΨΨ

−

k

j

i

A 1

0

0

0

��

�

…000 2

)1( Ψ−+Ψ+Ψ=Ψ ���nn

nn

17

Maximum Likelihood Estimation

• Mt motion at time t• Zt ={zt-nB … zt+nA} detection sets for frames t-nB to t+nA

• Random sampling:is an initial estimate of Mt, and refined using all the correct

detections

)()|(maxarg MMZpM tM

t δ=

)()|(maxarg~

SStM

MMZpMS

δ=

∏+

=++=

A

B

n

niSititSt ypMZp )|()|( ,z

Classical observation model

18

• The shaft position can be estimated when it is notdetected, with very good accuracy:

• Using the next frames makes the tracker:– More robust– More accurate– Almost Automatic!

Advantages

19

Other resultssame parameters

20

Making the tracker more robust

• Using a global motion model

21

upswing

downswing

Temporal Segmentation of the Sequence

22

Trajectory EstimationSeparated global motions models for– upswing– downswing

expressed in polar coordinates system

π

7π/3 7π/3

−π/2

23

Simple polynomial functions of rather small degrees

Upswingdeg. 4 polynomial

Downswingdeg. 6 polynomial

24

Trajectory Estimation

Same algorithm than before, with a global motion model•Robust estimation (b)•Refinement using the complete support (c)

25

Experimental results

26

Further Research

• Define a better global motion model (PCA ?)

• Analysis of the motion parameters

• Tracking of the golfer body