RTSP 2017EURASIP TUTORIAL DAYJuly 10, 2017, Bucharest
4 CAMERAS Unsupervised video orchestration based on Aesthetic features
Alessandro Neri, Federico Colangelo,
Federica Battisti. , Marco Carli
RTSP 2017EURASIP TUTORIAL DAYJuly 10, 2017, Bucharest
Outline
OUTLINE
MOTIVATION
01
02
VIDEO AESTHETICS
03
MULTI-OBJECTIVE OPTIMIZATION
04
EXPERIMENTAL RESULTS
05
CONCLUSION
06
RTSP 2017EURASIP TUTORIAL DAYJuly 10, 2017, Bucharest
Problem statement
Input: N synchronized cameras, shooting the same event E
Output: Editing script for the event E (e.g. from second 2 to 4 use camera 3)
RTSP 2017EURASIP TUTORIAL DAYJuly 10, 2017, Bucharest
Approach
Key Idea
combine multiple
camera
contributions based
on their aesthetic
value
Video Editing
Automatic cut based on
Aesthetic criteria
Dynamic aspects exploiting
the temporal information on
video
RTSP 2017EURASIP TUTORIAL DAYJuly 10, 2017, Bucharest
Problem Statement
Key Idea
combine multiple
camera
contributions based
on their aesthetic
value
Video Editing
Automatic cut based on
Aesthetic criteria
Dynamic aspects exploiting
the temporal information on
video
RTSP 2017EURASIP TUTORIAL DAYJuly 10, 2017, Bucharest
Frame-level aesthetic features
1
SimplicityUncluttered images have more aestheticvalue
2
ColorfulnessColor distribution
3
Sharpness
4
PatternTexture and Shapes
5
Composition1/3 rule
RTSP 2017EURASIP TUTORIAL DAYJuly 10, 2017, Bucharest
Frame-level aesthetic features
1
SimplicityUncluttered images have more aestheticvalue
2
ColorfulnessColor distribution
3
SharpnessColor distribution
4
PatternTexture and Shapes
5
Composition1/3 rule
RTSP 2017EURASIP TUTORIAL DAYJuly 10, 2017, Bucharest
Frame-level aesthetic features
1
SimplicityUncluttered images have more aestheticvalue
2
ColorfulnessColor distribution
3
Sharpness
4
PatternTexture and Shapes
5
Composition1/3 rule
RTSP 2017EURASIP TUTORIAL DAYJuly 10, 2017, Bucharest
Frame-level aesthetic features
1
SimplicityUncluttered images have more aestheticvalue
2
ColorfulnessColor distribution
3
Sharpness
4
PatternTexture and Shapes
5
Composition1/3 rule
RTSP 2017EURASIP TUTORIAL DAYJuly 10, 2017, Bucharest
AESTHETIC SCORE
Use of a bank of classifiers, each tuned to a specific feature group
Aesthetic value should be evaluated taking the context into account
1Train Classifiers per type of content
2Apply Classifiers
CUHKPQ dataset17690 images7 categories of subjects (animals, architecture, human, landscape, night, plant, static)Rated as high or low quality based on subjective experiments
FeatureExtraction
#1
FeatureExtraction
#2
FeatureExtraction
#3
FeatureExtraction
#4
FeatureExtraction
#5
SVM #1
SVM #2
SVM #3
SVM #4
BlockAveraging
SVM #5
BlockAveraging
BlockAveraging
BlockAveraging
BlockAveraging
RTSP 2017EURASIP TUTORIAL DAYJuly 10, 2017, Bucharest
Orchestration
For each interval, select the camera with the highest aesthetic value
What if there is a tie between cameras?
1
Partition the timeline into fixed time-length intervals (e.g. 2 sec)
Problem: tradeoff between dynamism and user confusion
2
Resolve ties based on the temporal
information
TI = maxš”ššš{šš šššš(š¹š š, š ā š¹šā1 š, š )}
Key idea: more dynamic shots tend to
be less boring for the viewers
3
RTSP 2017EURASIP TUTORIAL DAYJuly 10, 2017, Bucharest
Results
Random Orchestration Aesthetic Criterion
Randomlyorchestrated
Aesthetic-Criterion basedorchestration
MOS 3 4.1
ConfidenceInterval
0.7 0.5
RTSP 2017EURASIP TUTORIAL DAYJuly 10, 2017, Bucharest
MULTI-OBJECTIVE ORCHESTRATION
Problem:
ā¢ The aesthetic score of a shot can consistently outclass the others
āŖ E.g. more expert the operators ā¦ better the view of the sceneā¦
āŖ The video can become too static, favoring a single camera
ā¢ Can we make temporal segmentation dynamic?
ā¢ Can we leverage traditional video editing theory?
use multi-objective optimization and introduce a second function to penalize camera re-use
Multi objective optimization through the genetic
algorithm
RTSP 2017EURASIP TUTORIAL DAYJuly 10, 2017, Bucharest
Cutting Patterns
CONVENTIONAL
1
Wide Shot
2
Medium Shot
3
Close-Up
begins with the wide shot and then cuts to a medium shot and finally a close-up, working closer towards the subject or character
Professional directors use a combination of different shots types to keep the audience interested.
RTSP 2017EURASIP TUTORIAL DAYJuly 10, 2017, Bucharest
Cutting Patterns
REVEAL
1
Wide Shot
2
Medium Shot
3
Close-Up
Start with a tight shot and then use progressively wider shots to supply context, with a variety of angles by moving the camera around the subject.
Professional directors use a combination of different shots types to keep the audience interested.
RTSP 2017EURASIP TUTORIAL DAYJuly 10, 2017, Bucharest
Video editing
To evaluate the effectiveness of the editing with respect to the story telling a set of features is extracted from the shot sequence:
ā¢ shot type,
ā¢ camera angleāŖ Eye-level-angle: the point of view is put on the same footing with the subject
āŖ High angle: the camera looks down upon the subject
āŖ Low angle: the camera is looking up to the subject
āŖ Birdās-eye view: the camera provides an elevated view of the subject from above
āŖ Wormās-eye view; the camera gives a view of the subject from below.
ā¢ camera positionāŖ frontal view,
āŖ three quarter view,
āŖ side view,
āŖ back view
ā¢ camera movement.
RTSP 2017EURASIP TUTORIAL DAYJuly 10, 2017, Bucharest
Markov chains
Markovian approachā¢ :
Model the state of the system as a combination of camera attributes1.
Shot type, camera angle, camera position, camera movementāŖ
Assign transition probabilities according to video editing principles2.
ā¢ Smooth step between shot sizesā¢ Change position as well as shot size
(30 degrees rule)ā¢ Shoot in opposite directionsā¢ Stay on one side of an imaginary lineā¢ Vary pacing to create moods or
atmospheresā¢ Select shot length by how much
information it conveys.
MEDIUM0Ā°
CLOSE-UP0Ā°
WIDE0Ā°
MEDIUM30Ā°
CLOSE-UP30Ā°
WIDE30Ā°
MEDIUM60Ā°
CLOSE-UP60Ā°
WIDE60Ā°
0.01
0.45
0.15
0.14
0.15
0.10
RTSP 2017EURASIP TUTORIAL DAYJuly 10, 2017, Bucharest
MULTI-OBJECTIVE ORCHETRATION
Content Analysis
AestheticsScore
Assignment
Multi Objective
Opt.
Fine PacingTuning
ā¢ Randomly select the starting population (i.e. a set of random editings)1
2
3
4
5
RTSP 2017EURASIP TUTORIAL DAYJuly 10, 2017, Bucharest
MULTI-OBJECTIVE ORCHETRATION
Content Analysis
AestheticsScore
Assignment
Multi Objective
Opt.
Fine PacingTuning
ā¢ Randomly select the starting population (i.e. a set of random editings)
ā¢ Determine the fitness based on the diversity and the aesthetic value
1
2
3
4
5
RTSP 2017EURASIP TUTORIAL DAYJuly 10, 2017, Bucharest
MULTI-OBJECTIVE ORCHETRATION
Content Analysis
AestheticsScore
Assignment
Multi Objective
Opt.
Fine PacingTuning
ā¢ Randomly select the starting population (i.e. a set of random editings)
ā¢ Determine the fitness based on the diversity and the aesthetic value
ā¢ Obtain a set of Pareto-Optimal proposed editingsā¢ Re-use penalty: Kullback-Leiber divergence between the empirical probability
distribution induced by the proposed camera selection and the uniform distribution ā¢ Use a different set of aesthetic features to achieve better performances
ā¢ Lo, K. Y., Liu, K. H., & Chen, C. S. (2012, November). Assessment of photo aesthetics with efficiency. In Pattern Recognition (ICPR), 2012 21st International Conference on (pp. 2186-2189). IEEE.
1
2
3
4
5
RTSP 2017EURASIP TUTORIAL DAYJuly 10, 2017, Bucharest
MULTI-OBJECTIVE ORCHETRATION
Content Analysis
AestheticsScore
Assignment
Multi Objective
Opt.
Fine PacingTuning
ā¢ Randomly select the starting population (i.e. a set of random editings)
ā¢ Determine the fitness based on the diversity and the aesthetic value
ā¢ Obtain a set of Pareto-Optimal proposed editings
ā¢ Estimate content density of each shot through edge density
1
2
3
4
5
RTSP 2017EURASIP TUTORIAL DAYJuly 10, 2017, Bucharest
MULTI-OBJECTIVE ORCHETRATION
Content Analysis
AestheticsScore
Assignment
Multi Objective
Opt.
Fine PacingTuning
ā¢ Randomly select the starting population (i.e. a set of random editings)
ā¢ Determine the fitness based on the diversity and the aesthetic value
ā¢ Obtain a set of Pareto-Optimal proposed editings
ā¢ Estimate content density of each shot through edge density
ā¢ Apply Fine Pacing Tuning: subtract time from the less dense shots to give more camera to richer scenes
1
2
3
4
5
RTSP 2017EURASIP TUTORIAL DAYJuly 10, 2017, Bucharest
aesthetic features Revised
1
Colorfulness(f1)
2
Layout Composition(f2-f5)Distances of H, S, V, H+S+V distributionsfrom templates,
3
Edge Composition(f6-f9)
4
Global Texture(f10-f17)
5
General features(f18-f24)Blur, contrast, non-zero elements of HSV histogram
RTSP 2017EURASIP TUTORIAL DAYJuly 10, 2017, Bucharest
Results
Random Orchestration Aesthetic Criterion
RTSP 2017EURASIP TUTORIAL DAYJuly 10, 2017, Bucharest
Future works
ā¢ Neural style transfer uses deep neural networks to transform a picture
according to a given style
Train a deep neural network to transfer the editing style on the video based on video from Masters
RTSP 2017EURASIP TUTORIAL DAYJuly 10, 2017, Bucharest
Future works
ā¢ Imposing constraint on MAX time duration
āŖ Emphasis on VIDEO SUMMARIZATION
āŖ Jointly MAXIMIZE
o Aesthetics
o Representativeness
o Diversity
o Interestingness
o Importance
RTSP 2017EURASIP TUTORIAL DAYJuly 10, 2017, Bucharest
Conclusions
ā¢ The experimental results can be considered the basis for an
in-depth analysis of the proposed approach.
ā¢ On the other side, the computational cost of feature vector
extraction limits the use of this system to non real time
scenarios.
ā¢ Future versions of the system will focus on obtaining
automatic segmentation using boundary detection algorithms
and motion-based summarization, as well as on using feature
sets characterized by limited computational overhead, for
allowing a real-time implementation.
RTSP 2017EURASIP TUTORIAL DAYJuly 10, 2017, Bucharest
THANKS
The End
RTSP 2017EURASIP TUTORIAL DAYJuly 10, 2017, Bucharest
References
ā¢ [1] V. Mezaris E. Mavridaki, āA comprehensive aesthetic quality assessment method for natural images using basic rules of photography,ā in Proceedings of IEEE International Conference on Image Processing, (ICIP 2015), 2015, pp. 887 ā 891.
ā¢ [2] Lo, K. Y., Liu, K. H., & Chen, C. S. (2012, November). Assessment of photo aesthetics with efficiency. In Pattern Recognition (ICPR), 2012 21st International Conference on (pp. 2186-2189). IEEE.
ā¢ [3] B. Gong, W. Chao, K. Grauman, and F. Sha, āDiverse sequential subset selection for supervised video summarization,ā in Proceedings of the Neural Information Processing Systems (NIPS), 2014.
ā¢ [4] R. Kaiser, P. Torres, and M. Ho Ģffernig, āThe interaction ontology: Low- level cue processing in real-time group conversations,ā in 2nd ACM International Workshop on Events in Multimedia. EiMM ā10, ACM.
ā¢ [5] W. Taylor and F. Z. Qureshi, āAutomatic video editing for sensor-rich videos,ā in 2016 IEEE Winter Conference on Applications of Computer Vision (WACV), March 2016.
ā¢ [6] E. S. d. Lima, B. Feij, A. L. Furtado, A. Ciarlini, and C. Pozzer, āAutomatic video editing for video-based interactive storytelling,ā in 2012 IEEE International Conference on Multimedia and Expo, July 2012, pp. 806ā811.
ā¢ [7] K. Dancyger, The Technique of Film and Video Editing History, Theory, and Practice.
ā¢ [8] A. K. Moorthy, P. Obrador, and N. Oliver, āTowards computational models of the visual aesthetic appeal of consumer videos,ā in Proc. of Computer Vision, ECCV 2010.