Robust video fingerprinting systemRobust video fingerprinting system
Daniel Pereira [email protected] Loyola [email protected]
Confidential 22
Robust video fingerprinting system
• Summary
• Purpose of the system
• What is video fingerprinting
• Practical problems to solve
• Proposed solution
• Results analysis
Confidential 33
Robust video fingerprinting system
• Purpose of the system
• SkillUpJapan distributes digital contents
• FujiTV, TV Tokyo, SkyPerfecTV, Warner Brothers Japan, …
• Our platform, Uliza, is an extensible digital content
management system
• Piracy and DRM are of importance to digital contents
rights holders
Confidential 44
Robust video fingerprinting system
• Video fingerprinting
• A way to effectively tie a video, or a segment of it, to a
unique hash value
• Information needs to be stored and searched efficiently
• Avoid to store original contents provided by clients
• Contents should not be recreated from said fingerprint
Confidential 55
Robust video fingerprinting system
• Key technical aspects about video
• Measured characteristics
• Luma and chroma (brightness and color components)
• Edge detection, gradient orientation
• Time variance
• A movie is, after all, a sequence of images that change over time at
a defined rate
• Amount of data per frame
Confidential 66
Robust video fingerprinting system
• Efficiency metrics
• Uniqueness
• Accurately find videos we search; not return videos that
are not what we search
• Database
• Efficiently index the results in a database
• Solution must be fast
• Find the clip among many other videos in fastest time
Confidential 77
Robust video fingerprinting system
• Some practical problems to solve
• Current solutions have relatively accurate algorithms,
however the process is computationally expensive
• Partitioning of frames, complex algorithm
• Database storage is not taken seriously
• It is unaffordable to store information for every frame, or large
arrays of information for each fingerprint
• Slow search times when database grows
Confidential 88
Robust video fingerprinting system
• Proposed solution
• Lowers the needed resources and process time, while improving upon
results (Luma and time based indexes)
• Addresses algorithmic complexity by using simple methods (Euclidean
distance of vectors and Tanimoto correlation)
• Stores information in an efficient way, allowing for quick retrievals,
with use of Look-up Tables
Confidential 99
Robust video fingerprinting system
• Proposed solution (video properties):• Average the Luma value of each frame
• Luma values show prolonged, relatively constant, values that can be indexed to an interval of time
Lum
a
Time
ThresholdLum
a
Time
Confidential 1010
Robust video fingerprinting system
• Proposed solution (video properties):
• Average Luma calculated according to
Lum
a
Duration [start-time, end-
time[ (secs)
Average Luma calculated in T
seconds
[2, 8[ 102
[8,13[ 107
[13,14[ 198
[14,19[ 85
[590,600] 48
……
Time
€
ST =1
w × h × NL[x,y, f ]
y=0
h
∑x=0
w
∑f =0
N −1
∑
Confidential 1111
Robust video fingerprinting system
• Proposed solution (database):
• Using those indexes to store only segments we can save lots of space
• Each segment of several seconds has a value of 2 bytes
• Luma values range from 0 to 255
• Look-up table for segments
2.0
Luma
120
10.3 140
16.0
19.4
85
110
21.4 120
Time
Luma
time2.0 10.3 16.019.4
120
85
140
21.4
Confidential 1212
Robust video fingerprinting system
• Proposed solution (database):
4
Luma
10
15 160
19
32
125
13
… …
Time
2
Luma
120
5 140
9
12
85
110
… …
Time
2
Luma
120
5 140
9
12
85
110
… …
TimeComparisons
Fingerprints on database Fingerprint A
…
Fingerprint A
Fingerprint 1
Fingerprint 3
Time
Confidential 1313
Robust video fingerprinting system
• Proposed solution (algorithm):
• Tanimoto
• Tanimoto makes a correlation between C and the remaining elements outside C
• Euclidean vector distance
A BC
€
D = (Ai − Bi )2
i=0
n
∑
€
T(A,B) =A⋅ B
A2
+ B2
− A⋅ B
Confidential 1414
Robust video fingerprinting system
• Proposed solution (algorithm):
• Hierarchical approach
1. Look-up Table of segments
2. Compares the time indexes
3. & 4. Tanimoto Correlation and Vector Distance of Luma
• Look-up Tables discard perceptually different movies efficiently
• Comparison of time indexes also behaves efficiently
• The number of movies that are ultimately analyzed with Tanimoto
Correlation and Euclidean Vector Distance is very low
Confidential 1515
Robust video fingerprinting system
• Evaluation of algorithm:
• 220 movies were analyzed with each other
• Quality varies from FullHD to SD
• Duration ranges from 15 second commercials to full length movies
• Frame-rate of movies varies from 15fps to 30fps
• Comparison against C.G.O. (Centroids of Gradient Orientation) [1]
• Tests were conducted by searching scenes of 10 seconds
• Evaluation compares algorithm, database size and robustness of solutions
[1] Sunil Lee and Chang D. Yoo, “Robust Video Fingerprinting for Content-Based Video Identification”,
IEEE Trans. Circuits and Systems for Video Technology, vol. 18, no. 7, pp983-988, July, 2008
Confidential 1616
Robust video fingerprinting system
• Obtained results (database size):
SUJ CGO
DB Threshold Size (Kbytes) Size (Kbytes)
1 4,944
107,900
2 3,390
3 2,611
5 1,783
10 1,020
15 678
20 512
25 412
30 348
Confidential 1717
Robust video fingerprinting system
• Obtained results (run-time):
SUJ CGO
DB Threshold Time (Seconds) Time (Seconds)
1 74.16
92.41
2 31.56
3 21.13
5 10.77
10 8.84
15 8.66
20 10.1
25 12.42
30 15.11
Confidential 1818
Robust video fingerprinting system
• Obtained results (robustness):
SUJ CGO
DB threshold
False Positives
False Negatives
False Positives
False Negatives
1 8 0
211 1
2 12 0
3 9 0
5 11 0
10 11 3
15 8 8
20 9 6
25 11 14
30 9 18
Confidential 2020
Robust video fingerprinting system
• Summary
• State of the art solutions need to better address
practical issues
• The proposed algorithm can improve upon state of
the art algorithms on storage and speed of analysis
• Evaluation shows that the proposed solution also
provides higher robustness
Confidential 2121
Robust video fingerprinting system
Questions?
Daniel Pereira [email protected] Loyola [email protected]
Confidential 22
Robust video fingerprinting systemRobust video fingerprinting system
Daniel Pereira [email protected] Loyola [email protected]