Upload
others
View
10
Download
0
Embed Size (px)
Citation preview
[email protected] 1/16/2013 Lightning Talk: Local Binary Patterns
Introduction to Local Binary Patterns
Marc Norvig Principal Electronics Engineer
Embedded Video @ FLIR Systems, Inc.
[email protected] 1/16/2013 Lightning Talk: Local Binary Patterns
Three Questions
• What are Local Binary Patterns?
• Why should I care?
• Where can I use them?
[email protected] 1/16/2013 Lightning Talk: Local Binary Patterns
What are Local Binary Patterns?
• Computationally efficient texture descriptor
– Texture: • “Variation of data at scales smaller than the scale of interest” *
– Descriptor: • Compact representation that captures at least one essential
characteristic of the sample under examination
* (Petrou and Garcia Seville 2006, p. 1)
[email protected] 1/16/2013 Lightning Talk: Local Binary Patterns
Why should I care?
• Rare technique that is good, fast and cheap
– Good • Competitive with state-of-the-art for many image processing tasks
• Robust to monotonic changes in illumination
(http://docs.opencv.org/modules/contrib/doc/facerec/facerec_tutorial.html #id22)
[email protected] 1/16/2013 Lightning Talk: Local Binary Patterns
Why should I care?
• Rare technique that is good, fast and cheap
– Good • Competitive with state-of-the-art for many image processing tasks
• Robust to monotonic changes in illumination
– Fast • O(n)
– Cheap • Integer math, no multiply or divide
• No parameter tuning or image pre-processing required
• Compact descriptor (good performance with < 64 element vector)
[email protected] 1/16/2013 Lightning Talk: Local Binary Patterns
Where can I use LBP?
• Facial Analysis – Face Recognition – Face Detection – Expression Recognition – Gender Classification
• Texture Analysis – Classification – Segmentation – Background Subtraction – Visual Inspection
Your Problem Here
• Image Analysis – Interest Region Description – Image Forensics – Image Retrieval – Biometrics
• Motion Analysis – Gesture Recognition – Lip Reading – Object Detection – Gaze Tracking
[email protected] 1/16/2013 Lightning Talk: Local Binary Patterns
50
31 88
67
46
34
22
55 59
Implementation
Identify all pixels with luminance less than center pixel luminance
• 22 < 50
• 31 < 50
• 34 < 50
• 46 < 50
Remaining pixels are greater than or equal to center pixel luminance
• 55 > 50
• 59 > 50
• 67 > 50
• 88 > 50
Next, Binarize results
[email protected] 1/16/2013 Lightning Talk: Local Binary Patterns
0 1
1
0
0
0
1 1
Implementation
• Invariant to global illumination change
• Invariant to local contrast magnitude
• Captures gradient sign which describes the brighter/darker “shape” of the pattern
• Since contrast is orthogonal, it can be an effective additional feature in some cases
[email protected] 1/16/2013 Lightning Talk: Local Binary Patterns
1 10 00 101
1 10 00 101 binary
139 decimal
00
0
0
1 1
1
1
Implementation
Unroll the circular pattern to get LBP label
[email protected] 1/16/2013 Lightning Talk: Local Binary Patterns
Implementation details
Circular pattern doesn’t fit standard square sampling grid very well
[email protected] 1/16/2013 Lightning Talk: Local Binary Patterns
Implementation details
On-grid points use sample data directly
[email protected] 1/16/2013 Lightning Talk: Local Binary Patterns
Implementation details
Off-grid points use interpolation
[email protected] 1/16/2013 Lightning Talk: Local Binary Patterns
Implementation details
Nearest Neighbor is often sufficient
[email protected] 1/16/2013 Lightning Talk: Local Binary Patterns
• Calculate LBP for every pixel in exemplar
• Histogram is model of micro-textures in exemplar
Texture Classification: Training
LBP label
Freq
uen
cy
Model
[email protected] 1/16/2013 Lightning Talk: Local Binary Patterns
• Create LBP histogram from sample
• Find distance between sample & model histograms
Texture Classification: Recognition
Sample
LBP label
Freq
uen
cy
LBP label
Freq
uen
cy
Model
[email protected] 1/16/2013 Lightning Talk: Local Binary Patterns
• Three commonly used distance measures
– Histogram Intersection
– Log-likelihood
– Chi-squared
Texture Classification: Recognition
χ2 𝑆,𝑀 = (𝑆𝑏 − 𝑀𝑏)
2
𝑆𝑏 +𝑀𝑏
𝐵
𝑏=1
𝐷 𝑆,𝑀 = min (𝑆𝑏 𝑀𝑏)
𝐵
𝑏=1
𝐿 𝑆,𝑀 = − 𝑆𝑏 log𝑀𝑏
𝐵
𝑏=1
[email protected] 1/16/2013 Lightning Talk: Local Binary Patterns
• Machine learning finds threshold for classification
Texture Classification: Recognition
Sample1
LBP label
Freq
uen
cy
LBP label
Freq
uen
cy
Model
𝐷 𝑆,𝑀 = 0.93
Sample2 𝐷 𝑆,𝑀 = 0.28
LBP label
Freq
uen
cy
[email protected] 1/16/2013 Lightning Talk: Local Binary Patterns
Dimensionality Reduction • 256 bin histogram → 256 dimension feature
– Promised: Fewer than 64 dimensions
[email protected] 1/16/2013 Lightning Talk: Local Binary Patterns
Dimensionality Reduction • Uniform Local Binary Patterns
– Patterns with at most two contiguous regions
• Two patterns with one contiguous region
• Seven basic patterns with two contiguous regions
• Each basic pattern has eight orientations
[email protected] 1/16/2013 Lightning Talk: Local Binary Patterns
Dimensionality Reduction • 58 Uniform Local Binary Patterns plus one
OTHERS
[email protected] 1/16/2013 Lightning Talk: Local Binary Patterns
Dimensionality Reduction • Uniform Local Binary Patterns
– Reduces feature vector from 256 to 59 elements
• Helps with curse of dimensionality
– Natural images are ≈ 90% Uniform LBP
– Statistically more robust
• Produces better recognition in many applications
• Non-uniform patterns may not be robust to noise
[email protected] 1/16/2013 Lightning Talk: Local Binary Patterns
• Not robust to rotation - histogram comparison fails
Dealing with Rotation
(Ahonen, Matas, He, Pietikäinen 2009)
[email protected] 1/16/2013 Lightning Talk: Local Binary Patterns
• Solution: Use Fourier Transform of histogram
Dealing with Rotation
(Ahonen, Matas, He, Pietikäinen 2009)
[email protected] 1/16/2013 Lightning Talk: Local Binary Patterns
• Divide face into blocks, concatenate LBP histograms
Face Recognition: Face Descriptor
(Ahonen, Hadid, Pietikäinen 2006)
[email protected] 1/16/2013 Lightning Talk: Local Binary Patterns
• Compare test Subject to target Descriptor with Chi-squared distance measure:
j = image block, i = histogram bin
• Block-by-block weighting can improve performance
Face Recognition: Identification
(Ahonen, Hadid, Pietikäinen 2006)
χ2 𝑆, 𝐷 = (𝑆𝑖,𝑗 − 𝐷𝑖.𝑗)
2
𝑆𝑖,𝑗 + 𝐷𝑖,𝑗 𝑗,𝑖
χ2𝑤 𝑆, 𝐷 = 𝑤𝑗(𝑆𝑖,𝑗 − 𝐷𝑖.𝑗)
2
𝑆𝑖,𝑗 + 𝐷𝑖,𝑗 𝑗,𝑖
[email protected] 1/16/2013 Lightning Talk: Local Binary Patterns
• Block weights example, 7x7 grid of blocks
Face Recognition: Identification
(Ahonen, Hadid, Pietikäinen 2006)
[email protected] 1/16/2013 Lightning Talk: Local Binary Patterns
• Results for FERET database
Face Recognition: Identification
(Ahonen, Hadid, Pietikäinen 2006)
Expression Variation Illumination Variation
[email protected] 1/16/2013 Lightning Talk: Local Binary Patterns
• Hierarchical splitting
Unsupervised Segmentation
(Ojala, Pietikäinen 1999)
[email protected] 1/16/2013 Lightning Talk: Local Binary Patterns
• Agglomerative merging
Unsupervised Segmentation
(Ojala, Pietikäinen 1999)
[email protected] 1/16/2013 Lightning Talk: Local Binary Patterns
• Pixelwise classification
Unsupervised Segmentation
(Ojala, Pietikäinen 1999)
[email protected] 1/16/2013 Lightning Talk: Local Binary Patterns
• Natural image example
Unsupervised Segmentation
(Ojala, Pietikäinen 1999)
[email protected] 1/16/2013 Lightning Talk: Local Binary Patterns
Lipreading
(Zhao, Barnard, Pietikäinen 2009)
[email protected] 1/16/2013 Lightning Talk: Local Binary Patterns
Lipreading
(Zhao, Barnard, Pietikäinen 2009)
[email protected] 1/16/2013 Lightning Talk: Local Binary Patterns
Computer Vision using Local Binary Patterns Springer 2011
University of Oulu (Finland) Center for Machine Vision Research:
http://www.cse.oulu.fi/CMV
LBP Face Recognition available in OpenCV
Many implementations at MATLAB Central
http://www.mathworks.com/matlabcentral/
Resources
(Ojala, Pietikäinen 1999)
[email protected] 1/16/2013 Lightning Talk: Local Binary Patterns
• Ahonen T, Hadad A, Pietikäinen M. Face description with local binary pattens: application to face recognition. IEEE transactions on Pattern Analysis and Machine Intelligence 28. 2006:2037-2041
• Ahonen T, Matas J, He C, Pietikäinen M. Rotation invariant image description with local binary pattern histogram fourier features. Image Analysis, SCIA Proceedings, Lecture Notes in Computer Science 5575. 2009:61-70
• Ojala T, Pietikäinen M. Unsupervised texture segmentation using feature distribution. Pattern Recognition 32. 1999:477-478
• Petrou M, Garcia-Sevilla P. Image Processing: Dealing with Texture. Wiley; 2006
• Zhao G, Bernard M, Pietikäinen M. Lipreading with local spatiotemporal descriptors. IEEE Transactions on Multimedia vol. 11 no. 7. 2009:1254-1265
References