Upload
truongdieu
View
214
Download
2
Embed Size (px)
Citation preview
07/21/20173rd International Workshop on Computer Vision in Sports (CVsports) at CVPR 2017
SingletsMulti-resolution Motion Singularities
for Soccer Video AbstractionKaty Blanc, Diane Lingrand, Frederic Precioso
I3S Laboratory, Sophia Antipolis, France
1
Katy Blanc Diane Lingrand Frederic Precioso
Overview
07/21/20173rd International Workshop on Computer Vision in Sports (CVsports) at CVPR 2017
2
VIDEO & MOTIONSINGULARITIES
& SINGLETS
SOCCER SALIENT MOMENTS
Video Analysis
07/21/20173rd International Workshop on Computer Vision in Sports (CVsports) at CVPR 2017
3
Burst of video content production
new sources of videosbig databases : Youtube 8M [1]much analyzed types : meeting/conferences, movies, news and sports
Diverse applications :
browsing in database, automatic video surveillance, driverless car, …
Exponential amount of information:
a match of soccer of HDTV → 324000 images of 1920×1080 pixels each
Collaboration with Wildmoka, themselves in collaboration with l’INA and BeIN.
Related works: Video description
07/21/20173rd International Workshop on Computer Vision in Sports (CVsports) at CVPR 2017
4
Handcrafted features: Stip [3], iDT [4]
Deep learning representations [5]
Modelling human motion: [2]
Related works : sport abstraction
5
Clue Detection to detect highlights: ground color, jersey color, shot segmentation and view classification
Goals, attacks and other events usinglogo and score appearances and goal mouth position [7] Zawbaa et al.
Line marks positions and shot detection [6] Ye et al.
Face and skin detection, whistle detector and user specifications [8] Raventos et al.
07/21/20173rd International Workshop on Computer Vision in Sports (CVsports) at CVPR 2017
Overview
07/21/20173rd International Workshop on Computer Vision in Sports (CVsports) at CVPR 2017
6
VIDEO & MOTIONSINGULARITIES
& SINGLETS
SOCCER SALIENT MOMENTS
Inspiration: fluid movement
7
Inspired from the work of Druon et al. [9] and the further work of Kihl et al. [10]
07/21/20173rd International Workshop on Computer Vision in Sports (CVsports) at CVPR 2017
Optical Flow Approximation
07/21/20173rd International Workshop on Computer Vision in Sports (CVsports) at CVPR 2017
8
Optical Flow = discrete bivariable vector field
Polynomial subspace and the Legendre Basis
𝐹 ∶ 𝛺 → ℝ2
𝑥1 , 𝑥2 → 𝑈 𝑥1 , 𝑥2 , 𝑉 𝑥1 , 𝑥2
with 𝛺 = −1, 1 2
𝑃𝐾,𝐿 𝑥1, 𝑥2 =
𝑘=0
𝐾
𝑙=0
𝐿
𝛾𝑘,𝑙 . 𝑥1𝑘𝑥2
𝑙
with 𝐾 + 𝐿 < 𝐷
Projection on the Legendre Basis
𝑈 = 𝑢0,0𝑃0,0 + 𝑢0,1𝑃0,1 + 𝑢1,0𝑃1,0𝑉 = 𝑣0,0𝑃0,0 + 𝑣0,1𝑃0,1 + 𝑣1,0𝑃1,0
Polynomial projection
07/21/20173rd International Workshop on Computer Vision in Sports (CVsports) at CVPR 2017
9
Original Flow F
= (
Flow U Flow V Projection
𝑈𝑉
=0 0.02
3.91 0.𝑥1𝑥2
+0.69
−0.006
) ≃,
𝑈𝑉
= 1.38 + 0 + 0.007 + 0.023 + 0 − 4.47
−0.01 + 4.53 + 0.01 + 0 + 0 − 7𝐸−5
Approximation and coefficient analysis
10
From a simple analysis example to production, scenarization, event sementization
07/21/20173rd International Workshop on Computer Vision in Sports (CVsports) at CVPR 2017
𝑈𝑉
= 1.38
0
Frame number
Coefficient value
𝑢0,0 horizontal global displacement𝑣0,0 vertical global displacement
𝑢0,0 horizontal global position
𝑣0,0 vertical global position
counterattack
Singularity
07/21/20173rd International Workshop on Computer Vision in Sports (CVsports) at CVPR 2017
11
6 types of singularities
Then on the canonical basis𝑈𝑉
= 𝐴.𝑥1𝑥2
+ 𝑏 with 𝐴 ∈ 𝑀2,2 and 𝑏 ∈ ℝ2
λ1 and λ2 the Eigenvalues of A
𝑈 = 𝑢0,0𝑃0,0 + 𝑢0,1𝑃0,1 + 𝑢1,0𝑃1,0𝑉 = 𝑣0,0𝑃0,0 + 𝑣0,1𝑃0,1 + 𝑣1,0𝑃1,0
First projection on the Legendre basis
∆ 𝐴 = 𝑡𝑟 𝐴 2 − 4. det 𝐴 ,
Singularities extraction
07/21/20173rd International Workshop on Computer Vision in Sports (CVsports) at CVPR 2017
12
From a multi-resolution analysis of the optical flow … 𝑈𝑉
= 𝐴.𝑥1𝑥2
+ 𝑏
Singlets: match singularities during time
07/21/20173rd International Workshop on Computer Vision in Sports (CVsports) at CVPR 2017
13
From singularity on optical flow frame to tracks of singularities: Singlets
Overview
07/21/20173rd International Workshop on Computer Vision in Sports (CVsports) at CVPR 2017
14
VIDEO & MOTIONSINGULARITIES
& SINGLETS
SOCCER SALIENT MOMENTS
Application on soccer abstraction
07/21/20173rd International Workshop on Computer Vision in Sports (CVsports) at CVPR 2017
15
Our database
Zoom
Slow Motion
Global Excitement
Soccer Saliant Moment
Soccer Summarization: Our database
07/21/20173rd International Workshop on Computer Vision in Sports (CVsports) at CVPR 2017
16
Germany vs Portugal Nigeria vs Argentina France vs Honduras Switzerland vs France
4-0 2-3 3-0 2-5
© http://www.fifa.com/worldcup/matches/
Lack of standard benchmark for comparison sake…
07/21/20173rd International Workshop on Computer Vision in Sports (CVsports) at CVPR 2017
17
Soccer Summarization: Our database
Zoom detection
07/21/20173rd International Workshop on Computer Vision in Sports (CVsports) at CVPR 2017
18
Zoom motion is a pure star node singularity.Zoom combined with a translation is an improper node
Conditions: there is a singularity it is a star or improper node : ∆ 𝐴 = 0 time consistent : last for one second
Positives eigenvalues -> zoom out
Negatives eigenvalues -> zoom in
Determine the zoom direction center
Zoom detection
07/21/20173rd International Workshop on Computer Vision in Sports (CVsports) at CVPR 2017
19
• ∆ 𝐴 = 0 ↔ ∆ 𝐴 ≤ 0.2
• Threshold on an average |∆ 𝐴 | on a second
set to 0,2
• Comparison
- Global motion estimation [6,11]
- Duan et al. method [12] :Two histograms, one on magnitude one on angles to detect diagonal pattern (DL)
Slow Motion Detection
07/21/20173rd International Workshop on Computer Vision in Sports (CVsports) at CVPR 2017
20
How to differentiate a fast motion that has been artificially slowed down from a slow motion?
Slow Motion Detection
07/21/20173rd International Workshop on Computer Vision in Sports (CVsports) at CVPR 2017
21
Slow MotionFast Motion
Global excitement
07/21/20173rd International Workshop on Computer Vision in Sports (CVsports) at CVPR 2017
22
Spatial histogram of 3x3 per frameSum spatial histograms on 10 framesThreshold set on 1500 to select the most agitated moments
Soccer Saliant Moment
07/21/20173rd International Workshop on Computer Vision in Sports (CVsports) at CVPR 2017
23
Within 30 seconds:- at least two zoom direction changes- an activity peaks higher than 1500 in the farthest view- a slow motion replay in a close up view
Example of Summarization
07/21/20173rd International Workshop on Computer Vision in Sports (CVsports) at CVPR 2017
24
Zooms Agitation Slow Motion
Extension to other sports
07/21/20173rd International Workshop on Computer Vision in Sports (CVsports) at CVPR 2017
25
Set and Trained on Soccer and Tested on HandballAll hyperparameters, parameters and SVM were trained and set on soccer videos and we test ourframework on 5 minutes of Qatar Handball World Championship final without any adjustement or
retraining.
Conclusion
07/21/20173rd International Workshop on Computer Vision in Sports (CVsports) at CVPR 2017
26
Optical Flow Approximation
Possibility to use :
- different basis- different degree- other space than the polynomial one
Singlets
Track singularities
Length histograms
Compute a description like for IDT
Singularity extraction
Vanishing point
6 types for the affine approximation
Motion description
Global distribution
Sport Video Abstraction
Zoom detection
Slow Motion Detection
Global excitement
Because there is not only soccer in life
07/21/20173rd International Workshop on Computer Vision in Sports (CVsports) at CVPR 2017
27
Abstraction of concert
Facial Emotion
Action recognition
07/21/20173rd International Workshop on Computer Vision in Sports (CVsports) at CVPR 2017
28
3rd International Workshop on Computer Vision in Sports: CVSports
References
07/21/20173rd International Workshop on Computer Vision in Sports (CVsports) at CVPR 2017
29
[1] S. Abu-El-Haija, N. Kothari, J. Lee, P. Natsev, G. Toderici, B. Varadarajan, and S. ijayanarasimhan. Youtube-8m: A large-scalevideo classification benchmark. CoRR, abs/1609.08675, 2016.
[2] Gorelick, Lena, et al. "Actions as space-time shapes." IEEE transactions on pattern analysis and machine intelligence 29.12 (2007): 2247-2253.
[3] I. Laptev. On space-time interest points. Int. J. Comput. Vision, 64(2-3):107–123, Sept. 2005.
[4] H. Wang, A. Kläser, C. Schmid, and C.-L. Liu. Action Recognition by Dense Trajectories. In IEEE Conference on Computer Vision & Pattern Recognition, pages 3169–3176, Colorado Springs, United States, June 2011.
[5] D. Tran, L. Bourdev, R. Fergus, L. Torresani, and M. Paluri. Learning spatiotemporal features with 3d convolutional net-works. In roceedings of the IEEE International Conference on Computer Vision, pages 4489–4497, 2015.
[6] Q. Ye, Q. Huang, W. Gao, and S. Jiang. Exciting event detection in broadcast soccer video with mid-level description and incremental learning. In Proceedings of the 13th annual ACM international conference on Multimedia, 2005.
[7] H. M. Zawbaa, N. El-Bendary, A. E. Hassanien, and T. Kim. Event detection based approach for soccer video summarization using machine learning. International Journal of Multimedia and Ubiquitous Engineering, 2012
[8] A. Raventos, R. Quijada, L. Torres, and F. Tarres. Automatic summarization of soccer highlights using audio-visual descriptors. arXiv preprint arXiv:1411.6496, 2014.
[9] Druon, Martin. Modélisation du mouvement par polynômes orthogonaux: application à l'étude d'écoulements fluides. Diss. Université de Poitiers, 2009.
[10] O. Kihl, B. Tremblais, and B. Augereau. Multivariate orthogonal polynomials to extract singular points. In IEEE International Conference on Image Processing 2008. ICIP 2008 San Diego, CA, United States, Oct. 2008.
[11] X. Qian. Global Motion Estimation and Its Applications. INTECH Open Access Publisher, 2012.
[12] L.-Y. Duan, M. Xu, Q. Tian, C.-S. Xu, and J. S. Jin. A unified framework for semantic shot classification in sports video. IEEE Transactions on Multimedia, 2005
Optical Flow Extraction
07/21/20173rd International Workshop on Computer Vision in Sports (CVsports) at CVPR 2017
30
Gunnar Farneback MethodPixel Neighborhood Approximation
Approximation translation
Relations
Speed vector estimation
Polynomial Space Projection
07/21/20173rd International Workshop on Computer Vision in Sports (CVsports) at CVPR 2017
31
Scalar product
Legendre basis with 𝑤 𝑥1, 𝑥2 = 1 and Ω = −1,1 2
Polynomial degree 𝐷 ⇒ 𝑛𝑑 =(𝐷+1)(𝐷+2)
2polynomials in the basis and so 𝑛𝑑
coefficients
Flot optique blurring (decreasing with the degree)
Multi-Resolution Singularity
07/21/20173rd International Workshop on Computer Vision in Sports (CVsports) at CVPR 2017
32
One sliding windows -> possibly one singularity
The same singularity can be at different size and aprroximately the sameposition: We keep the one with less angular deviation from the original flow.
𝑑𝑒𝑣 =
𝜔∈Ω
1
2sin 𝜃 𝜔 − 𝜃′ 𝜔
With 𝜃 𝜔 the angle of the motion vector at the pixel position 𝜔 for the originalf flow and 𝜃′ 𝜔 for the polynomial approximation