OpenCV IntroductionHang Xiao
Oct 26, 2012
History 1999 Jan : lanched by Intel, real time machine vision library for UI, optimized code
for intel
2000 Jun : OpenCV alpha 3。 2000 Dec : OpenCV beta 1 for linux
2006 : the first 1.0 version supports Mac OS
2008 mid : obtain corporate support from Willow Garage
2009 Sep : OpenCV 1.2( beta2.0
2009 Oct : Version 2.0 released。 2010 Dec : OpenCV 2.2。 2011 Aug : OpenCV 2.3。 2012 Apr : OpenCV 2.4.
Overview Goals
Develop a universal toolbox for research and development in the field of Computer Vision
Algorithms More than 350 algorithms, 500 API
Programming language C/C++, C#, Ch , Python, Ruby, Matlab, and Java (using JavaCV)
OS support Windows, Android, Maemo, FreeBSD, OpenBSD, iOS, Linux and Mac
OS.
Licence BSDlisence, free for commercial and non-commmercial
Overview - Applications 2D and 3D feature toolkits Egomotion estimation Facial recognition system Gesture recognition Human–computer interaction (HCI) Mobile robotics Motion understanding Object identification Segmentation and Recognition Stereopsis Stereo vision: depth perception from 2 cameras Structure from motion (SFM)Motion tracking
Overview - A statistical machine learning library Boosting (meta-algorithm)
Decision tree learning
Gradient boosting trees
Expectation-maximization algorithm
k-nearest neighbor algorithm
Naive Bayes classifier
Artificial neural networks
Random forest
Support vector machine (SVM)
Outline
Image Analysis Structural Analysis Object Recognition Motion Analysis and Object Tracking 3D Reconstruction
Outline
Image Analysis Structural Analysis Object Recognition Motion Analysis and Object Tracking 3D Reconstruction
Image Analysis Thresholds Statistics Pyramids Morphology Distance transform Flood fill Feature detection Contours retrieving
Image Thresholding
Fixed threshold; Adaptive
threshold;
Image Thresholding Examples
Source picture Fixed threshold Adaptive threshold
Statistics
min, max, mean value, standard deviation over the image
Norms C, L1, L2 Multidimensional histograms Spatial moments up to order 3 (central,
normalized, Hu)
Multidimensional Histograms Histogram operations calculation, normalization, comparison, back project
Histograms types: Dense histograms Signatures (balanced tree)
EMD algorithm The EMD computes the distance between two distributions, which are
represented by signatures. The signatures are sets of weighted features that capture the distributions.
The features can be of any type and in any number of dimensions, and are defined by the user.
The EMD is defined as the minimum amount of work needed to change one signature into the other
EMD – a method for the histograms comparison
.),(
,
,),(
),(
,1,,1,
,
,
jiji
ij
jiij
jijiij
ji
qandpelementsthebetweendistancetheqpd
tscoefficienweightf
f
qpdfQPEMD
historamstwoQjQqPiPp
Image Pyramids
Gaussian and Laplacian pyramids
Image segmentation by pyramids
Image Pyramids
Gaussian and Laplacian
Pyramid-based color segmentation
On still pictures And on movies
Morphological Operations Two basic morphology operations using structuring element:
erosion dilation
More complex morphology operations: opening : erosion + dilation closing : dilation + erosion morphological gradient : the difference between the dilation and the erosion of an
image top hat : the difference between an input image and its opening black hat : the difference between the closing and its input image
Morphological Operations Examples Morphology - applying Min-Max. Filters and its combinations
Opening IoB= (IB)BDilatation IBErosion IBImage I
Closing I•B= (IB)B TopHat(I)= I - (IB) BlackHat(I)= (IB) - IGrad(I)= (IB)-(IB)
Distance Transform
Calculate the distance for all non-feature points to the closest feature point
Two-pass algorithm, 3x3 and 5x5 masks, various metrics predefined
Flood Filling
Simple Gradient
Feature Detection Fixed filters (Sobel operator, Laplacian); Optimal filter kernels with floating point coefficients
(first, second derivatives, Laplacian) Special feature detection (corners) Canny operator Hough transform (find lines and line segments) Gradient runs
Canny Edge Detector
Hough TransformDetects lines in a binary image
• Probabilistic Hough Transform• Standard Hough
Transform
Another Sample of the Hough Transform Using
Source picture Result
Contour Retrieving
The contour representation: Chain code (Freeman code) Polygonal representation
Initial PointChain code for the curve: 34445670007654443
Contour representation
Hierarchical representation of contours
Image Boundary
(W1) (W2) (W3)
(B2) (B3) (B4)
(W5) (W6)
Contours Examples
Source Picture(300x600 = 180000 pts total)
Retrieved Contours (<1800 pts total)
After Approximation(<180 pts total)
And it is rather fast: ~70 FPS for 640x480 on complex scenes
Outline
Image Analysis Structural Analysis Object Recognition Motion Analysis and Object Tracking 3D Reconstruction
Structural Analysis Contours processing Approximation Hierarchical representation Shape characteristics Matching
Geometry Contour properties Fitting with primitives PGH: pair-wise geometrical histogram for the
contour.
Contour Processing
Approximation: RLE algorithm (chain code) Teh-Chin approximation (polygonal) Douglas-Peucker approximation (polygonal);
Contour moments (central and normalized up to order 3) Hierarchical representation of contours Matching of contours
Hierarchical Representation of Contours
A contour is represented with a binary tree
Given the binary tree, the contour can be retrieved with arbitrary precision
The binary tree is quasi invariant to translations, rotations and scaling
Contours matching
Matching based on hierarchical representation of contours
Geometry
Properties of contours: (perimeter, area, convex hull, convexity defects, rectangle of minimum area)
Fitting: (2D line, 3D line, circle, ellipse) Pair-wise geometrical histogram
Pair-wise geometrical histogram (PGH)
),( jip
.),(/),()(
,),(/),()(
,)](),2(),1(),(),2(),1([
iic
jjr
TcccrrrPGH
jipjipijE
jipjipjiE
MEEENEEEf
Outline
Image Analysis Structural Analysis Object Recognition Motion Analysis and Object Tracking 3D Reconstruction
Object Recognition Eigen objects Hidden Markov Models
Eigen Objects
..},,,{
,,1},,,,{
),(/1
,/1),()(
,cov}{,,1,},,{
21
111
1
21
vectorseigenandvalueseigenand
basiseigenmmieeee
uuve
umuuuuuc
matrixariancecCnmmiRuuuu
im
iiii
in
iii
lk
kl
iki
il
m
k
klll
jl
ll
ilij
ij
nin
iii
Eigen objects (continued)
Hidden Markov Model
Definitions
},...,,{ 21 NsssS },...,,{ 21 MoooO
Oqt )}|(\{ 1 itjtijij sqsqPaaA
)}|(\{ jiijij soPbbB
)}(\{ 0 iii sqP
- The set of states
- The set of measurements
- The state at time t
- The transition probability matrix
- The conditional probability matrix
- The starting states distribution
Embedded HMM for Face Recognition
Model-
- Face ROI partition
Face recognition using Hidden Markov Models
One person – one HMMStage 1 – Train every HMM
Stage 2 – Recognition
Pi - probability
Choose max(Pi)
…1
n
i
Outline
Image Analysis Structural Analysis Object Recognition Motion Analysis and Object Tracking 3D Reconstruction
Motion Analysis and Object Tracking
Background subtraction Motion templates Optical flow Active contours Estimators
Background Subtraction
Background model (normal distribution) Background statistics functions:
Average Standard deviation Running average
Motion Templates
Object silhouette Motion history images Motion history gradients Motion segmentation algorithm
silhouette MHIMHG
Motion Segmentation Algorithm
Two-pass algorithm labeling all motion segments
Motion Templates Example
Motion templates allow to retrieve the dynamic characteristics of the moving object
Optical Flow Block matching technique Horn & Schunck technique Lucas & Kanade technique Pyramidal LK algorithm 6DOF (6 degree of freedom) algorithm
y
xt
yyx
yxx
II
IbIII
IIIGyxX
bXG
dtdyyIdtdxxItItyxIdttdyydxxI
,,
,),,(
,
);/(/)/(//);,,(),,(
2
2
Optical flow equations:
Pyramidal Implementation of the optical flow algorithm
J image I image
Image Pyramid Representation
Iterative Lucas – Kanade Scheme
Generic Image
(L-1)-th Level
L-th Level
Location of point u on image uL=u/2L
Spatial gradient matrix
Standard Lucas – Kanade scheme for optical flow computation at level L dL
Guess for next pyramid level L – 1
Finally,
Image pyramid building
Optical flow computation
2
2
,
,
yyx
yxx
III
IIIG
)(21 LLL dgg
00 gdd
dUV
6DOF Algorithm
N
i ROI
Tit
N
i ROIi
Ti IIdsII
sXIsII
i11
///
).(sX
Parametrical optical flow equations:
Active Contours
Snake energy: Internal energy: External energy: Two external energy types:
extEEE int
curvcont EEE int
conimgext EEE
min
,)(
,
imgcurvcont
img
img
EEEE
IgradE
IE
Estimators
Kalman filter ConDensation filter
Kalman object tracker
Outline
Image Analysis Structural Analysis Object Recognition Motion Analysis and Object Tracking 3D Reconstruction
3D reconstruction
Camera Calibration View Morphing POSIT
Camera Calibration Define intrinsic and extrinsic camera parameters. Define Distortion parameters
],[],,,[,,,
100
00
,][
3
2
1
333231
232221
131211
vupZYXPttt
Trrrrrrrrr
Rcfcf
A
PRTAp
yy
xx
.
)],2/(2[)(~)],2/(2[)(~
222
212
42
21
221
42
21
yxr
yyrpxprkrkcvvv
xxrpyprkrkcuuu
y
x
Camera CalibrationNow, camera calibration can be done by holding checkerboard in front of the camera for a few seconds.
And after that you’ll get:3D view of etalon Un-distorted image
View Morphing
POSIT Algorithm
Perspective projection:
Weak-perspective projection:
iiiiii YZfyXZfx )/(,)/(
./,, ZfsYsyXsx iiii
OpenCV Websites http://opencv.org
OpenCV official webpage.
http://opencvlibrary.sourceforge.net/ OpenCV documentation and FAQs.
OpenCV Examples adaptiveskindetector : detect skin area fback_c : dense Franeback optical flow contours : calculate contours on different levels delaunay : delaunay triangle find_obj : SURF Detector and Descriptor using either FLANN or brute
force matching on planar objects morphology : open/close, erode/dilate motempl :motion templates mser_sample : Maximal Extremal Region interest point detector polar_transforms : illustrates Linear-Polar and Log-Polar image transforms pyramid_segmentation : color pyramid segmentation
Thanks !!!