View
214
Download
1
Category
Tags:
Preview:
Citation preview
Computer Vision GroupUniversity of California Berkeley
Visual Grouping and Object Recognition
Jitendra Malik*
U.C. Berkeley
* with S. Belongie, C. Fowlkes, T. Leung, D. Martin, G. Mori, J. Puzicha, J.Shi, X. Ren
Computer Vision GroupUniversity of California Berkeley
From images/video to objects
Labeled sets: tiger, grass etc
Computer Vision GroupUniversity of California Berkeley
Computer Vision GroupUniversity of California Berkeley
Computer Vision GroupUniversity of California Berkeley
Consistency
A
B C
• A,C are refinements of B• A,C are mutual refinements • A,B,C represent the same percept
• Attention accounts for differences
Image
BG L-bird R-bird
grass bush
headeye
beakfar body
headeye
beak body
Perceptual organization forms a tree:
Two segmentations are consistent when they can beexplained by the samesegmentation tree (i.e. theycould be derived from a single perceptual organization).
Computer Vision GroupUniversity of California Berkeley
Outline
• Finding boundaries
• Recognizing objects
• Recognizing actions
Computer Vision GroupUniversity of California Berkeley
Finding boundaries: Is texture a problem or a solution?
image orientation energy
Computer Vision GroupUniversity of California Berkeley
Statistically optimal contour detection
• Use humans to segment a large collection of natural images.
• Train a classifier for the contour/non-contour classification using orientation energy and texture gradient as features.
Computer Vision GroupUniversity of California Berkeley
Orientation Energy
• Gaussian 2nd derivative and its Hilbert pair
•
• Can detect combination of bar and edge features [Perona & Malik 90]
22 )()( evenodd fIfIOE
Computer Vision GroupUniversity of California Berkeley
Texture gradient = Chi square distance between texton histograms in half disks across edge
Texture gradient = Chi square distance between texton histograms in half disks across edge
i
j
k
K
m ji
jiji mhmh
mhmhhh
1
22
)()(
)]()([
2
1),(Chi-square
0.1
0.8
Computer Vision GroupUniversity of California Berkeley
Computer Vision GroupUniversity of California Berkeley
Computer Vision GroupUniversity of California Berkeley
ROC curve for local boundary detection
Computer Vision GroupUniversity of California Berkeley
Outline
• Finding boundaries
• Recognizing objects
• Recognizing actions
Computer Vision GroupUniversity of California Berkeley
Biological Shape
• D’Arcy Thompson: On Growth and Form, 1917– studied transformations between shapes of organisms
Computer Vision GroupUniversity of California Berkeley
Deformable Templates: Related Work
• Fischler & Elschlager (1973)
• Grenander et al. (1991)
• von der Malsburg (1993)
Computer Vision GroupUniversity of California Berkeley
Matching Framework
• Find correspondences between points on shape
• Fast pruning
• Estimate transformation & measure similarity
model target
...
Computer Vision GroupUniversity of California Berkeley
Comparing Pointsets
Computer Vision GroupUniversity of California Berkeley
Shape ContextCount the number of points inside each bin, e.g.:
Count = 4
Count = 10
...
Compact representation of distribution of points relative to each point
Computer Vision GroupUniversity of California Berkeley
Shape Context
Computer Vision GroupUniversity of California Berkeley
Comparing Shape Contexts
Compute matching costs using Chi Squared distance:
Recover correspondences by solving linear assignment problem with costs Cij
[Jonker & Volgenant 1987]
Computer Vision GroupUniversity of California Berkeley
Matching Framework
• Find correspondences between points on shape
• Fast pruning
• Estimate transformation & measure similarity
model target
...
Computer Vision GroupUniversity of California Berkeley
Fast pruning
• Find best match for the shape context at only a few random points and add up cost
),(minarg
),(),(
2*
*
1
2
ui
jqueryui
ij
query
r
jiquery
SCSCSC
SCSCSSdist
Computer Vision GroupUniversity of California Berkeley
Matching Framework
• Find correspondences between points on shape
• Fast pruning
• Estimate transformation & measure similarity
model target
...
Computer Vision GroupUniversity of California Berkeley
• 2D counterpart to cubic spline:
• Minimizes bending energy:
• Solve by inverting linear system
• Can be regularized when data is inexact
Thin Plate Spline Model
Duchon (1977), Meinguet (1979), Wahba (1991)
Computer Vision GroupUniversity of California Berkeley
MatchingExample
model target
Computer Vision GroupUniversity of California Berkeley
Outlier Test Example
Computer Vision GroupUniversity of California Berkeley
Object Recognition Experiments
• Handwritten digits
• COIL 3D objects (Nayar-Murase)
• Human body configurations
• Trademarks
Computer Vision GroupUniversity of California Berkeley
Terms in Similarity Score• Shape Context difference
• Local Image appearance difference– orientation– gray-level correlation in Gaussian window– … (many more possible)
• Bending energy
Computer Vision GroupUniversity of California Berkeley
Handwritten Digit Recognition
• MNIST 60 000: – linear: 12.0%
– 40 PCA+ quad: 3.3%
– 1000 RBF +linear: 3.6%
– K-NN: 5%
– K-NN (deskewed): 2.4%
– K-NN (tangent dist.): 1.1%
– SVM: 1.1%
– LeNet 5: 0.95%
• MNIST 600 000 (distortions): – LeNet 5: 0.8%– SVM: 0.8%– Boosted LeNet 4: 0.7%
• MNIST 20 000: – K-NN, Shape Context
matching: 0.63%
Computer Vision GroupUniversity of California Berkeley
Computer Vision GroupUniversity of California Berkeley
COIL Object Database
Computer Vision GroupUniversity of California Berkeley
Prototypes Selected for 2 Categories
Details in Belongie, Malik & Puzicha (NIPS2000)
Computer Vision GroupUniversity of California Berkeley
Error vs. Number of Views
Computer Vision GroupUniversity of California Berkeley
Human body configurations
Computer Vision GroupUniversity of California Berkeley
Deformable Matching
• Kinematic chain-based deformation model
• Use iterations of correspondence and deformation
• Keypoints on exemplars are deformed to locations on query image
Computer Vision GroupUniversity of California Berkeley
Results
Computer Vision GroupUniversity of California Berkeley
Trademark Similarity
Computer Vision GroupUniversity of California Berkeley
Recognizing objects in scenes
Computer Vision GroupUniversity of California Berkeley
Outline
• Finding boundaries
• Recognizing objects
• Recognizing actions
Computer Vision GroupUniversity of California Berkeley
Examples of Actions• Movement and posture change
– run, walk, crawl, jump, hop, swim, skate, sit, stand, kneel, lie, dance (various), …
• Object manipulation– pick, carry, hold, lift, throw, catch, push, pull, write, type, touch, hit,
press, stroke, shake, stir, turn, eat, drink, cut, stab, kick, point, drive, bike, insert, extract, juggle, play musical instrument (various)…
• Conversational gesture– point, …
• Sign Language
Computer Vision GroupUniversity of California Berkeley
Key cues for action recognition
• “Morpho-kinesics” of action (shape and movement of the body)
• Identity of the object/s
• Activity context
Computer Vision GroupUniversity of California Berkeley
Image/Video Stick figure Action
• Stick figures can be specified in a variety of ways or at various resolutions (deg of freedom)– 2D joint positions– 3D joint positions– Joint angles
• Complete representation
• Evidence that it is effectively computable
Computer Vision GroupUniversity of California Berkeley
Tracking by Repeated Finding
Computer Vision GroupUniversity of California Berkeley
Achievable goals in 3 years
• Reasonable competence at object recognition at crude category level (~1000)
• Detection/Tracking of humans as kinematic chains, assuming adequate resolution.
• Recognition of ~10-100 actions and compositions thereof.
Recommended