Upload
eunix
View
733
Download
1
Embed Size (px)
Citation preview
Automatic Photo Selection for Media and Entertainment
ApplicationsEkaterina Potapova,
Marta Egorova, Ilia Safonov
National Nuclear Research University MEPhI Moscow, Russia
GraphiCon 20095-9 October
Applications
Automatic Photo Selection for Media and Entertainment Applications
GraphiCon 2009 2
Applications
Automatic Photo Selection for Media and Entertainment Applications
GraphiCon 2009 2
Applications – photo book
Images are taken from printbook.ru, ehow.com, snapfish.com.au, smilebooks.co.ukGraphiCon 2009 3
Automatic Photo Selection for Media and Entertainment Applications
Applications – slide show
Photos from ITaS’2008GraphiCon 2009 4
Automatic Photo Selection for Media and Entertainment Applications
General workflow
GraphiCon 2009 5
Automatic Photo Selection for Media and Entertainment Applications
Detection of low-quality photos
GraphiCon 2009 5
Automatic Photo Selection for Media and Entertainment Applications
General workflow
General workflow
Detection of low-quality photos
Adaptive quantization on
time-camera plane
GraphiCon 2009 5
Automatic Photo Selection for Media and Entertainment Applications
Selection of appealing photos
Detection of low-quality photos
Adaptive quantization on
time-camera plane
General workflow
GraphiCon 2009 5
Automatic Photo Selection for Media and Entertainment Applications
Detection of low-quality photos
GraphiCon 2009 6
Automatic Photo Selection for Media and Entertainment Applications
Estimation of JPEG qualityA.Foi et al.,2007
3
1
3
1,9
1
i jjiqK
Images are taken from en.wikipedia.org
Quantization Table
GraphiCon 2009 7
Automatic Photo Selection for Media and Entertainment Applications
Detection of backlit, low-contrast & blurred photos
+
Good photo
Bad photo
True
False
…
…
11 TF
ii TF
NN TF
}{ iF
1w
iw
Nw
N
ii Tw
1
Two Ada Boost classifiers committee: -for detection of low-contrast and backlit photos-for detection of blurred photos
GraphiCon 2009 8
Automatic Photo Selection for Media and Entertainment Applications
Detection of backlit and low-contrast photos - 1 S1/S2 - ratio of tones in shadows to midtones
)/()(]85,0[
1 NMiHS )/()(]170,85(
2 NMiHS
GraphiCon 2009 9
Automatic Photo Selection for Media and Entertainment Applications
S11/S12 - ratio of tones in first to second part of shadows
)(/)(]42,0[
11 NMiHS )(/)(]85,42(
12 NMiHS
Detection of backlit and low-contrast photos - 1
GraphiCon 2009 9
Automatic Photo Selection for Media and Entertainment Applications
M1/M2 - ratio of the histogram maximum in shadows to the maximum in midtones
]255,0[]85,0[1 ))(max(/))(max( iHiHM
]255,0[]170,85(2 ))(max(/))(max( iHiHM
Detection of backlit and low-contrast photos - 1
GraphiCon 2009 9
Automatic Photo Selection for Media and Entertainment Applications
P1 - location of the histogram maximum in shadows
]85,0[1 ))(max()(| iHlHlP
]85,0[))(max( iH
P1
Detection of backlit and low-contrast photos - 1
GraphiCon 2009 9
Automatic Photo Selection for Media and Entertainment Applications
C – global contrastlowhighC
})][|min{},][|min(min{0
00
i
k
CkHiHiHilow })][|max{},][|max(max{1
11
ik
RR CkHiHiHihigh
H0
C0
C1
H1
Detection of backlit and low-contrast photos - 1
GraphiCon 2009 9
Automatic Photo Selection for Media and Entertainment Applications
Training set: 480 photos
Error rate on cross-validation test : ~0.055
Testing set: 1830 with 2% affected by backlit and low-contrast photos
The number of False Positives (FP) is 10 The number of False Negatives (FN) is 3
Low-contrast photoBacklit photo
Detection of backlit and low-contrast photos - 2
GraphiCon 2009 10
Automatic Photo Selection for Media and Entertainment Applications
Image
Intensity image
Z1=[-1 1]Z2=[-1 0 1]
Z3=[-1 0 0 1]
Z10=[-1 0 0 0 0 0 0 0 0 0 1]
I.Safonov et al.,2008
…
Edge image
ii ZIE
3
bgrI
Histogram
iHe
Normalized entropy
k
kiHeiA )1)(log(
Entropy to [0, 1]
iAn
121 An An F
10
12
iiAnF
23 A F
?
?
?
?
An An
GraphiCon 2009 11
Detection of blurred photosAutomatic Photo Selection for Media and Entertainment Applications
SDI
SVSDI F h )(
4
Crete et al., 2007
cr
crDISDI,
),(
crcrDBcrDIhh
hecrDBcrDISV
,)),(),((1001
1)),(),((
F.Crete et al.,2007
LPFIBh
HPFIDI
HPFBDB hh
231 An An F
11
22
iiAnF
23 A F
?
Image
Blurred image Edge image
Edge image
Comparison of the images
HPF=[1 -1]LPF=[1 1 1 1 1 1 1 1 1]/9
Detection of blurred photos
GraphiCon 2009 11
Automatic Photo Selection for Media and Entertainment Applications
Training set: 416 photos
Error rate on cross-validation test : ~0.07
Testing set: 1830 with 171 blurred photos
The number of False Positives (FP) is 34
The number of False Negatives (FN) is 10SDI
SVSDI F h )(
4
231 An An F
11
22
iiAnF
23 A F
Detection of blurred photos
GraphiCon 2009 11
Automatic Photo Selection for Media and Entertainment Applications
Time and camera-based quantization
Photo creation time
Photo source
1
243
L
H
evenisiiNpsH
oddisiiHYpi :)2/1(
:2/)1(
i is an index of source
L is time between the least and the most time for the largest source
Nps is a number of sources
H = L/M
M is count of images
Nregion < M
Calculation of bounding boxes
Partition into 2 app. equal subregions
Seeking for the biggest region
1200
3600
2400
72000 36000 T, s21600GraphiCon 2009 11
Automatic Photo Selection for Media and Entertainment Applications
Nregion < Ngroup
GraphiCon 2009 12
Automatic Photo Selection for Media and Entertainment Applications
Salient Photo SelectionThe most appealing photo is the most salient photo
L.Itti, C.Koch et al.
Images are taken from the Internet
Conspicuity maps
Gaussian pyramids
Image
Intensity image
r-channel g-channel b-channel
R-channel G-channel B-channel Y-channel
Orientation map
Intensity map
Color map
Saliency map
Feature maps
Gabor pyramids
GraphiCon 2009 13
Automatic Photo Selection for Media and Entertainment Applications
Salient Photo Selection
original image
saliency map
intensity map
color map
orientation map
ROI
Automatic Photo Selection for Media and Entertainment Applications
Salient Photo Selection
GraphiCon 2009 14Image is taken from the Internet
Automatic Photo Selection for Media and Entertainment Applications
Salient Photo Selection
GraphiCon 2009 15
Saliency Index
4),( maxS
yxS 124
88
11
100
81 9262
83105 70
83
11124
Automatic Photo Selection for Media and Entertainment Applications
Salient Photo Selection
GraphiCon 2009 15
Saliency Index
4),( maxS
yxS
81
88 62 92
105 70
100
Main Disadvantages:
average number of FP increases a lot with picture size
0
1
2
3
4
5
6
7
0 500 1000 1500 2000 2500
S, pixelav
gn
FP
0
5
10
15
20
25
30
0 500 1000 1500 2000 2500 3000
S, pixelt,
s.
0
1
2
3
4
5
6
7
0 500 1000 1500 2000 2500S, pixel
avg
nF
PBefore skin tone detection After skin tone detection
0
5
10
15
20
25
30
0 1000 2000 3000S, pixel
t, s.
After modification Before modification
We consider, that images of people attracts more attention
processing time also increases a lot with picture size
Six places were detected erroneously
Modifications: image down sampling is applied at preprocessing step
optimization of search using color information – skin tone detection
P.Viola, M.Jones, 2001
Automatic Photo Selection for Media and Entertainment Applications
Face Detection
GraphiCon 2009 16
Viola-Jones, Intel OpenCV
Before modifications After modifications
Photos ranking
Heuristic formula, experiments have shown that value w=25 gives the best result
Automatic Photo Selection for Media and Entertainment Applications
GraphiCon 2009 17
124
88
11
116 92
118148 95
62
100
118
62
Photos ranking
Heuristic formula, experiments have shown that value w=25 gives the best result
Automatic Photo Selection for Media and Entertainment Applications
GraphiCon 2009 17
124
88
11
100
116
92
148 95
Automatic Photo Selection for Media and Entertainment Applications
Results and discussion
GraphiCon 2009 18
Automatic Photo Selection for Media and Entertainment Applications
Results and discussion
GraphiCon 2009 18
Autocollage choiceOur choice
Automatic Photo Selection for Media and Entertainment Applications
Results and discussion
GraphiCon 2009 18
Automatic Photo Selection for Media and Entertainment Applications
Results and discussion
GraphiCon 2009 18
Autocollage choiceOur choice
Automatic Photo Selection for Media and Entertainment Applications
Results and discussion
GraphiCon 2009 18
Automatic Photo Selection for Media and Entertainment Applications
Results and discussion
GraphiCon 2009 18
Autocollage choiceOur choice
Automatic Photo Selection for Media and Entertainment Applications
Results and discussion
GraphiCon 2009 19
Set 1Set 1 Set 2Set 2 Set 3Set 3 Set 4Set 4 Set 5Set 5 SumSum
Agree with experts 6 5 6 5 7 29
Acceptable 3 4 4 4 2 17
Unacceptable 1 1 0 1 1 4
Agree with experts 2 2 2 6 5 17
Acceptable 6 7 7 0 4 24
Unacceptable 2 1 1 4 1 9
Agree with experts 2 2 3 4 4 15
Acceptable 5 5 4 2 5 21
Unacceptable 3 3 3 4 1 14
Prop
osed
Auto
Colla
geRa
ndom
?Automatic Photo Selection for Media and Entertainment Applications
Questions & Answers
GraphiCon 2009 8
Automatic Photo Selection for Media and Entertainment Applications
GraphiCon 2009 9
Thank you for your attention
=)