Computer Vision

CSPP 56553

Artificial Intelligence

March 3, 2004

Roadmap

• Motivation– Computer vision applications

• Is a Picture worth a thousand words?– Low level features

• Feature extraction: intensity, color

– High level features• Top-down constraint: shape from stereo, motion,..

• Case Study: Vision as Modern AI– Fast, robust face detection (Viola & Jones 2002)

Perception

• From observation to facts about world– Analogous to speech recognition– Stimulus (Percept) S, World W

• S = g(W)

– Recognition: Derive world from percept• W=g’(S)

• Is this possible?

Key Perception Problem

• Massive ambiguity– Optical illusions

• Occlusion

• Depth perception

• “Objects are closer than they appear”

• Is it full-sized or a miniature model?

Image Ambiguity

Handling Uncertainty

• Identify single perfect correct solution– Impossible!

• Noise, ambiguity, complexity

• Solution:– Probabilistic model– P(W|S) = αP(S|W) P(W)

• Maximize image probability and model probability

Handling Complexity

• Don’t solve the whole problem– Don’t recover every object/position/color…

• Solve restricted problem– Find all the faces– Recognize a person– Align two images

Modern Computer Vision Applications

• Face / Object detection

• Medical image registration

• Face recognition

• Object tracking

Vision Subsystems

Image Formation

Images and Representations

• Initially pixel images – Image as NxM matrix of pixel values

– Alternate image codings• Grey-scale intensity values

• Color encoding: intensities of RGB values

Images

Grey-scale Images

Color Images

Image Features

• Grey-scale and color intensities– Directly access image signal values

– Large number of measures• Possibly noisy

• Only care about intensities as cues to world

• Image Features:– Mid-level representation

– Extract from raw intensities

– Capture elements of interest for image understanding

Edge Detection

Edge Detection

• Find sharp demarcations in intensity• 1) Apply spatially oriented filters

• E.g. vertical, horizontal, diagonal

• 2) Label above-threshold pixels with edge orientation• 3) Combine edge segments with same orientation:

line

Top-down Constraints

• Goal: Extract objects from images– Approach: apply knowledge about how the world

works to identify coherent objects

Motion: Optical Flow

• Find correspondences in sequential images– Units which move

together represent objects

Stereo

Stereo Depth Resolution

Texture and Shading

Edge-Based 2-3D Reconstruction

Assume world of solid polyhedra with 3-edge verticesApply Waltz line labeling – via Constration Satisfaction

Basic Object Recognition

• Simple idea:– extract 3-D shapes from image– match against \shape library"

• Problems:– extracting curved surfaces from image– representing shape of extracted object– representing shape and variability of library object classes– improper segmentation, occlusion– unknown illumination, shadows, markings, noise, complexity, etc.

• Approaches:– index into library by measuring invariant properties of objects– alignment of image feature with projected library object feature– match image against multiple stored views (aspects) of library object– machine learning methods based on image statistics

Hand-written Digit Recognition

Summary

• Vision is hard:– Noise, ambiguity, complexity

• Prior knowledge is essential to constrain problem– Cohesion of objects, optics, object features

• Combine multiple cues– Motion, stereo, shading, texture,

• Image/object matching:– Library: features, lines, edges, etc

• Apply domain knowledge: Optics• Apply machine learning: NN, NN, CSP, etc

Computer Vision Case Study

• “Rapid Object Detection using a Boosted Cascade of Simple Features”, Viola/Jones ’01

• Challenge:– Object detection:

• Find all faces in an arbitrary images

– Real-time execution• 15 frames per second

– Need simple features, classifiers

Rapid Object Detection Overview

• Fast detection with simple local features– Simple fast feature extraction

• Small number of computations per pixel• Rectangular features

– Feature selection with Adaboost• Sequential feature refinement

– Cascade of classifiers• Increasingly complex classifiers• Repeatedly rule out non-object areas

Picking Features

• What cues do we use for object detection?– Not direct pixel intensities– Features

• Can encode task specific domain knowledge (bias)– Difficult to learn directly from data

– Reduce training set size

• Feature system can speed processing

Rectangle Features

• Treat rectangles as units– Derive statistics

• Two-rectangle features– Two similar rectangular regions

• Vertically or horizontally adjacent

– Sum pixels in each region• Compute difference between regions

Rectangle Features II

• Three-rectangle features– 3 similar rectangles: horizontally/vertically

• Sum outside rectangles

• Subtract from center region

• Four-rectangle features– Compute difference between diagonal pairs

• HUGE feature set: ~180,000

Rectangle Features

Computing Features Efficiently

• Fast detection requires fast feature calculation• Rapidly compute intermediate representation

– “Integral image”

– Value for point (x,y) is sum of pixels above, left

– ii(x,y) = Σx’<=x,y’<=y i(x,y)

– Computed by recurrence• s(x,y) = s(x,y-1) + i(x,y) , where s(x,y) cumulative row

• ii(x,y) = ii(x-1,y) + s(x,y)

• Compute rectangle sum with 4 array references

Rectangle Feature Summary

• Rectangle features– Relatively simple– Sensitive to bars, edges, simple structure

• Coarse

– Rich enough for effective learning– Efficiently computable

Learning an Image Classifier

• Supervised training: +/- examples• Many learning approaches possible• Adaboost:

– Selects features AND trains classifier– Improves performance of simple classifiers

• Guaranteed to converge exponentially rapidly

– Basic idea: Simple classifier• Boosts performance by focusing on previous errors

Feature Selection and Training

• Goal: Pick only useful features from 180000– Idea: Small number of features effective

• Learner selects single feature that best separates +/- ve examples– Learner selects optimal threshold for each feature– Classifier h(x) = 1 if pf(x)<pθ, 0 otherwise

Basic Learning Results

• Initial classification: Frontal faces– 200 features– Finds 95%, 1/14000 false positive– Very fast

• Adding features adds to computation time

• Features interpretable– Darker region around eyes that nose/cheeks– Eyes are darker than bridge of nose

Primary Features

“Attentional Cascade”

• Goal: Improved classification, reduced time– Insight: Small – fast – classifiers can reject

• But have very few false negatives– Reject majority of uninteresting regions quickly

– Focus computation on interesting regions

• Approach: “Degenerate” decision tree• Aka “cascade”

• Positive results passed to high detection classifiers– Negative results rejected immediately

Cascade Schematic

All Sub-window Features

CL 1 CL 2 CL 3

F F F

T T T MoreClassifiers

Reject Sub-Window

Cascade Construction

• Each stage is a trained classifier– Tune threshold to minimize false negatives– Good first stage classifier

• Two feature strong classifier – eye/check + eye/nose

• Tuned: Detect 100%; 40% false positives

– Very computationally efficient • 60 microprocessor instructions

Cascading

• Goal: Reject bad features quickly– Most features are bad

• Reject early in processing, little effort

– Good regions will trigger full cascade• Relatively rare

• Classification is progressively more difficult– Rejected the most obvious cases already

• Deeper classifiers more complex, more error-prone

Cascade Training

• Tradeoffs: Accuracy vs Cost– More accurate classifiers: more features, complex

– More features, more complex: Slower

– Difficult optimization

• Practical approach– Each stage reduces false positive rate

– Bound reduction in false pos, increase in miss

– Add features to each stage until meet target

– Add stages until overall effectiveness targets met

Results

• Task: Detect frontal upright faces– Face/non-face training images

• Face: ~5000 hand-labeled instances

• Non-face: ~9500 random web-crawl, hand-checked

– Classifier characteristics:• 38 layer cascade

• Increasing number of features: 1,10,25,… : 6061

– Classification: Average 10 features per window• Most rejected in first 2 layers

• Process 384x288 image in 0.067 secs

Detection Tuning

• Multiple detections:– Many subwindows around face will alert– Create disjoint subsets

• For overlapping boundaries, only report one – Return average of corners

• Voting:– 3 similarly trained detectors

• Majority rules

– Improves overall

Conclusions

• Fast, robust facial detection– Simple, easily computable features– Simple trained classifiers– Classification cascade allows early rejection

• Early classifiers also simple, fast

– Good overall classification in real-time

Some Results

Vision in Modern Ai

• Goals: – Robustness– Multidomain applicability– Automatic acquisition– Speed: Real time

• Approach:– Simple mechanisms, feature selection– Machine learning: Tune features, classification