Computer VisionMachine Learning

FeaturesPresented By

Dr. Keith Haynes

Introduction Appearance-Based Approach Features Classifiers Face Detection Walkthrough Questions

Outline

Computer vision is a field that includes methods for acquiring, processing, analyzing, and understanding images.

What does that mean? What are some computer vision task?

Computer Vision

Detection

Are there any faces in this image?

Recognition

+ ClassLabel

Database of Classes

TestSubject

Pre p ro c e ssing

Se nsing

Fe a ture Extra c tio n

C la ssific a tio n

Po st-Pro c e ssing

Dec isio n

Pattern Recognition System

Computer Vision a difficult problem

Images vary due to the relative camera-object pose

Frontal, profile, etc.

Pose

Components may vary in:◦ Size◦ Shape◦ Color◦ texture

Structural Components

Some objects have the ability to change shape

Deformability

Computational Complexity There are many possible objects Scale Orientation

15 15 32 44 57 84 138 219 244 248 248 248 248 246 244 242 223 222 233 244 245 223 160 749 14 36 50 57 81 119 128 208 244 250 248 251 221 153 145 158 191 209 228 217 177 133 6236 27 54 87 106 121 149 169 133 126 160 222 226 171 150 182 177 175 176 179 172 158 122 3527 56 100 124 144 155 144 147 86 42 64 165 190 152 188 212 173 162 187 198 196 174 110 4011 69 97 97 105 112 91 80 46 15 41 157 186 146 182 160 113 100 152 188 202 188 119 4022 52 41 31 29 28 36 43 5 2 52 173 187 122 135 79 47 19 52 90 131 168 142 3529 33 20 19 23 27 46 69 52 11 33 146 174 115 99 48 35 31 31 52 97 148 150 7417 41 48 72 104 122 153 237 235 56 33 162 242 175 73 91 113 152 181 197 201 192 167 13417 41 74 80 76 75 106 224 235 51 36 165 251 183 71 120 103 136 194 208 199 195 171 13029 54 94 107 101 94 122 212 119 30 46 168 251 225 167 148 141 125 175 190 180 176 154 11644 72 93 100 104 113 111 80 33 11 43 163 242 228 182 108 163 157 156 143 150 166 141 10750 99 142 126 108 110 79 10 52 37 54 166 243 229 194 140 163 157 155 147 140 132 111 9443 103 161 165 158 160 116 20 97 84 81 173 244 234 215 200 178 160 165 166 147 120 102 9433 84 142 191 224 234 185 53 125 110 76 160 240 223 194 211 202 184 171 164 154 137 119 10935 76 165 222 243 230 159 73 127 101 50 139 230 201 155 195 189 183 171 171 160 139 128 12242 89 186 230 231 177 62 25 27 61 100 159 196 191 178 167 135 153 165 183 173 146 141 12650 138 191 173 138 97 35 10 10 42 83 142 166 131 83 56 68 71 136 187 192 176 154 10838 133 116 83 78 64 29 14 15 61 119 182 189 135 82 51 50 54 66 148 198 184 167 11228 96 89 81 93 81 37 22 69 109 157 190 203 196 171 148 74 67 49 107 167 179 167 9326 77 127 157 160 114 34 13 77 150 200 209 215 229 224 197 52 40 68 94 129 165 151 7022 60 159 210 191 126 44 19 40 101 145 152 161 173 164 151 76 94 145 156 155 158 122 4114 33 134 187 170 122 71 47 33 53 91 106 125 144 131 140 171 207 227 232 207 154 86 126 18 74 122 143 128 85 71 77 113 164 185 204 226 225 227 235 234 239 235 196 125 49 10 12 39 67 111 131 95 95 96 121 127 168 212 224 225 232 245 241 245 243 175 72 19 0

Crux of the Problem

As the dimensions increase, the volume of the space increases exponentially

The data points occupy a volume that is mainly empty.

Under these conditions, tasks such as estimating a probability distribution function become very difficult.

In high dimensions the training sets may not provide adequate coverage of the space.

Curse of Dimensionality

Machine learning is the science of getting computers to act without being explicitly programmed.

Applications◦ self-driving cars◦ speech recognition◦ effective web search◦ understanding of the human genome

Machine Learning

15 15 32 44 57 84 138 219 244 248 248 248 248 246 244 242 223 222 233 244 245 223 160 749 14 36 50 57 81 119 128 208 244 250 248 251 221 153 145 158 191 209 228 217 177 133 6236 27 54 87 106 121 149 169 133 126 160 222 226 171 150 182 177 175 176 179 172 158 122 3527 56 100 124 144 155 144 147 86 42 64 165 190 152 188 212 173 162 187 198 196 174 110 4011 69 97 97 105 112 91 80 46 15 41 157 186 146 182 160 113 100 152 188 202 188 119 4022 52 41 31 29 28 36 43 5 2 52 173 187 122 135 79 47 19 52 90 131 168 142 3529 33 20 19 23 27 46 69 52 11 33 146 174 115 99 48 35 31 31 52 97 148 150 7417 41 48 72 104 122 153 237 235 56 33 162 242 175 73 91 113 152 181 197 201 192 167 13417 41 74 80 76 75 106 224 235 51 36 165 251 183 71 120 103 136 194 208 199 195 171 13029 54 94 107 101 94 122 212 119 30 46 168 251 225 167 148 141 125 175 190 180 176 154 11644 72 93 100 104 113 111 80 33 11 43 163 242 228 182 108 163 157 156 143 150 166 141 10750 99 142 126 108 110 79 10 52 37 54 166 243 229 194 140 163 157 155 147 140 132 111 9443 103 161 165 158 160 116 20 97 84 81 173 244 234 215 200 178 160 165 166 147 120 102 9433 84 142 191 224 234 185 53 125 110 76 160 240 223 194 211 202 184 171 164 154 137 119 10935 76 165 222 243 230 159 73 127 101 50 139 230 201 155 195 189 183 171 171 160 139 128 12242 89 186 230 231 177 62 25 27 61 100 159 196 191 178 167 135 153 165 183 173 146 141 12650 138 191 173 138 97 35 10 10 42 83 142 166 131 83 56 68 71 136 187 192 176 154 10838 133 116 83 78 64 29 14 15 61 119 182 189 135 82 51 50 54 66 148 198 184 167 11228 96 89 81 93 81 37 22 69 109 157 190 203 196 171 148 74 67 49 107 167 179 167 9326 77 127 157 160 114 34 13 77 150 200 209 215 229 224 197 52 40 68 94 129 165 151 7022 60 159 210 191 126 44 19 40 101 145 152 161 173 164 151 76 94 145 156 155 158 122 4114 33 134 187 170 122 71 47 33 53 91 106 125 144 131 140 171 207 227 232 207 154 86 126 18 74 122 143 128 85 71 77 113 164 185 204 226 225 227 235 234 239 235 196 125 49 10 12 39 67 111 131 95 95 96 121 127 168 212 224 225 232 245 241 245 243 175 72 19 0

Humans Don’t Understand

Model-Based◦ Uses 3D models to generate images◦ Original and rendered images compared for

classification Appearance-Based

◦ Learns how to classify image via training examples

Computer Vision Approaches

S e t o f D iscrim in ato ry

F eature s

Tra in ing S e t

L ea rn

P e rfo rmF eatu re E x trac tion

Test S ub ject

F eatureR epresen tatio n o f

Test S ub ject

C lass ify

C lassL ab el

Q u ick ly & A c cu ra te ly

The Concept

Features are learned through example images, usually known as a training set

3D Models are not needed Utilizes machine learning and statistical

analysis

Appearance-Based Approach

Training Sets

ORL (Face recognition)

COIL-100 (Object Recognition)

A feature is a calculation performed on a portion of an image that yields a number

Features are used to represent the entity being analyzed.

Image Features

15 15 32 44 57 84 138 219 244 248 248 248 248 246 244 242 223 222 233 244 245 223 160 749 14 36 50 57 81 119 128 208 244 250 248 251 221 153 145 158 191 209 228 217 177 133 6236 27 54 87 106 121 149 169 133 126 160 222 226 171 150 182 177 175 176 179 172 158 122 3527 56 100 124 144 155 144 147 86 42 64 165 190 152 188 212 173 162 187 198 196 174 110 4011 69 97 97 105 112 91 80 46 15 41 157 186 146 182 160 113 100 152 188 202 188 119 4022 52 41 31 29 28 36 43 5 2 52 173 187 122 135 79 47 19 52 90 131 168 142 3529 33 20 19 23 27 46 69 52 11 33 146 174 115 99 48 35 31 31 52 97 148 150 7417 41 48 72 104 122 153 237 235 56 33 162 242 175 73 91 113 152 181 197 201 192 167 13417 41 74 80 76 75 106 224 235 51 36 165 251 183 71 120 103 136 194 208 199 195 171 13029 54 94 107 101 94 122 212 119 30 46 168 251 225 167 148 141 125 175 190 180 176 154 11644 72 93 100 104 113 111 80 33 11 43 163 242 228 182 108 163 157 156 143 150 166 141 10750 99 142 126 108 110 79 10 52 37 54 166 243 229 194 140 163 157 155 147 140 132 111 9443 103 161 165 158 160 116 20 97 84 81 173 244 234 215 200 178 160 165 166 147 120 102 9433 84 142 191 224 234 185 53 125 110 76 160 240 223 194 211 202 184 171 164 154 137 119 10935 76 165 222 243 230 159 73 127 101 50 139 230 201 155 195 189 183 171 171 160 139 128 12242 89 186 230 231 177 62 25 27 61 100 159 196 191 178 167 135 153 165 183 173 146 141 12650 138 191 173 138 97 35 10 10 42 83 142 166 131 83 56 68 71 136 187 192 176 154 10838 133 116 83 78 64 29 14 15 61 119 182 189 135 82 51 50 54 66 148 198 184 167 11228 96 89 81 93 81 37 22 69 109 157 190 203 196 171 148 74 67 49 107 167 179 167 9326 77 127 157 160 114 34 13 77 150 200 209 215 229 224 197 52 40 68 94 129 165 151 7022 60 159 210 191 126 44 19 40 101 145 152 161 173 164 151 76 94 145 156 155 158 122 4114 33 134 187 170 122 71 47 33 53 91 106 125 144 131 140 171 207 227 232 207 154 86 126 18 74 122 143 128 85 71 77 113 164 185 204 226 225 227 235 234 239 235 196 125 49 10 12 39 67 111 131 95 95 96 121 127 168 212 224 225 232 245 241 245 243 175 72 19 0

Image

Haar Features Computes the

difference between sums of two or more areas

Edge detector

A B

D

C

15 15 32 44 57 84 138 219 244 248 248 248 248 246 244 242 223 222 233 244 245 223 160 749 14 36 50 57 81 119 128 208 244 250 248 251 221 153 145 158 191 209 228 217 177 133 6236 27 54 87 106 121 149 169 133 126 160 222 226 171 150 182 177 175 176 179 172 158 122 3527 56 100 124 144 155 144 147 86 42 64 165 190 152 188 212 173 162 187 198 196 174 110 4011 69 97 97 105 112 91 80 46 15 41 157 186 146 182 160 113 100 152 188 202 188 119 4022 52 41 31 29 28 36 43 5 2 52 173 187 122 135 79 47 19 52 90 131 168 142 3529 33 20 19 23 27 46 69 52 11 33 146 174 115 99 48 35 31 31 52 97 148 150 7417 41 48 72 104 122 153 237 235 56 33 162 242 175 73 91 113 152 181 197 201 192 167 13417 41 74 80 76 75 106 224 235 51 36 165 251 183 71 120 103 136 194 208 199 195 171 13029 54 94 107 101 94 122 212 119 30 46 168 251 225 167 148 141 125 175 190 180 176 154 11644 72 93 100 104 113 111 80 33 11 43 163 242 228 182 108 163 157 156 143 150 166 141 10750 99 142 126 108 110 79 10 52 37 54 166 243 229 194 140 163 157 155 147 140 132 111 9443 103 161 165 158 160 116 20 97 84 81 173 244 234 215 200 178 160 165 166 147 120 102 9433 84 142 191 224 234 185 53 125 110 76 160 240 223 194 211 202 184 171 164 154 137 119 10935 76 165 222 243 230 159 73 127 101 50 139 230 201 155 195 189 183 171 171 160 139 128 12242 89 186 230 231 177 62 25 27 61 100 159 196 191 178 167 135 153 165 183 173 146 141 12650 138 191 173 138 97 35 10 10 42 83 142 166 131 83 56 68 71 136 187 192 176 154 10838 133 116 83 78 64 29 14 15 61 119 182 189 135 82 51 50 54 66 148 198 184 167 11228 96 89 81 93 81 37 22 69 109 157 190 203 196 171 148 74 67 49 107 167 179 167 9326 77 127 157 160 114 34 13 77 150 200 209 215 229 224 197 52 40 68 94 129 165 151 7022 60 159 210 191 126 44 19 40 101 145 152 161 173 164 151 76 94 145 156 155 158 122 4114 33 134 187 170 122 71 47 33 53 91 106 125 144 131 140 171 207 227 232 207 154 86 126 18 74 122 143 128 85 71 77 113 164 185 204 226 225 227 235 234 239 235 196 125 49 10 12 39 67 111 131 95 95 96 121 127 168 212 224 225 232 245 241 245 243 175 72 19 0

Image199 527

-328

414152

262

+ -

Feature representation is determined by:◦ the task being performed◦ performance constraints such as accuracy and

calculation time.

Two Groups◦ Global – feature uses the entire image◦ Local – feature uses parts of the image

Image Features

Local Features Attempts to identify the critical areas from

a set of images for class discrimination How are critical areas identified? Requires an exhaustive search of possible

sub-windows

Rectangular Features

2MP image has 922,944,480,000 possible features and took 16.45 min

Height Width Possible Features Time1 1 1 < ms2 2 9 < ms3 3 36 < ms4 4 100 < ms

24 24 90,000 0.2 ms128 128 68,161,536 0.1 sec256 256 1,082,146,816 1.19 sec512 512 17,247,043,584 18.48 sec

A single Haar feature is a weak classifier

A set of features can form a strong classifier

Features and Classification

Haar Feature Effectiveness

Features in Set Number of Sets1 90,0002 8,099,910,0003 7.28976E+144 6.56056E+195 5.90424E+246 5.31352E+297 4.78185E+348 4.30333E+399 3.87266E+44

10 3.48504E+49

Sets of Features

Exhaustive Search◦ For 5 features 5.9x1024 unique sets

Find best features one at a time.◦ Find the first best feature◦ Find the feature that works best with the first

feature, and so on◦ For 5 features 449,990 sets searched

Increase step size

Search Method

Example Features

Together they form a strong classifier

15 15 32 44 57 84 138 219 244 248 248 248 248 246 244 242 223 222 233 244 245 223 160 749 14 36 50 57 81 119 128 208 244 250 248 251 221 153 145 158 191 209 228 217 177 133 6236 27 54 87 106 121 149 169 133 126 160 222 226 171 150 182 177 175 176 179 172 158 122 3527 56 100 124 144 155 144 147 86 42 64 165 190 152 188 212 173 162 187 198 196 174 110 4011 69 97 97 105 112 91 80 46 15 41 157 186 146 182 160 113 100 152 188 202 188 119 4022 52 41 31 29 28 36 43 5 2 52 173 187 122 135 79 47 19 52 90 131 168 142 3529 33 20 19 23 27 46 69 52 11 33 146 174 115 99 48 35 31 31 52 97 148 150 7417 41 48 72 104 122 153 237 235 56 33 162 242 175 73 91 113 152 181 197 201 192 167 13417 41 74 80 76 75 106 224 235 51 36 165 251 183 71 120 103 136 194 208 199 195 171 13029 54 94 107 101 94 122 212 119 30 46 168 251 225 167 148 141 125 175 190 180 176 154 11644 72 93 100 104 113 111 80 33 11 43 163 242 228 182 108 163 157 156 143 150 166 141 10750 99 142 126 108 110 79 10 52 37 54 166 243 229 194 140 163 157 155 147 140 132 111 9443 103 161 165 158 160 116 20 97 84 81 173 244 234 215 200 178 160 165 166 147 120 102 9433 84 142 191 224 234 185 53 125 110 76 160 240 223 194 211 202 184 171 164 154 137 119 10935 76 165 222 243 230 159 73 127 101 50 139 230 201 155 195 189 183 171 171 160 139 128 12242 89 186 230 231 177 62 25 27 61 100 159 196 191 178 167 135 153 165 183 173 146 141 12650 138 191 173 138 97 35 10 10 42 83 142 166 131 83 56 68 71 136 187 192 176 154 10838 133 116 83 78 64 29 14 15 61 119 182 189 135 82 51 50 54 66 148 198 184 167 11228 96 89 81 93 81 37 22 69 109 157 190 203 196 171 148 74 67 49 107 167 179 167 9326 77 127 157 160 114 34 13 77 150 200 209 215 229 224 197 52 40 68 94 129 165 151 7022 60 159 210 191 126 44 19 40 101 145 152 161 173 164 151 76 94 145 156 155 158 122 4114 33 134 187 170 122 71 47 33 53 91 106 125 144 131 140 171 207 227 232 207 154 86 126 18 74 122 143 128 85 71 77 113 164 185 204 226 225 227 235 234 239 235 196 125 49 10 12 39 67 111 131 95 95 96 121 127 168 212 224 225 232 245 241 245 243 175 72 19 0

Feature Extraction

Original Image

47-229-498179106-157-3461124-99-257423

FeatureSet

Feature selection is important, is application dependent

Statistical methods very useful with high dimensionality

Local identify discriminating areas or features images

No universal solution Features can be combined

Summary

The Classifier

Types of Classifiers Linear Discriminant Analysis Fisher Discriminant Analysis Bayesian Classifier Neural Networks K-Nearest Neighbor Classifier

Features can used to form a coordinate space called the feature space.

Euclidean distance is used as the metric

Nearest Neighbor Classification

221

21211 )(...)( dd xxxxX

Feature Selection

The distance is not used directly for feature selection

The higher the ratio, the better the filter

In order to prevent one class from dominating, an exponential function was used

The sum of function for all test images was used for selection

[Liu, Srivastava, Gallivan]

Feature Space Examples

Separation and grouping Better Classification Low Classification Rates

Rapid Classification Tree

“Divide and Conquer” Instead of trying to solve a difficult problem

all at once, divide it into several parts Each of the resulting parts should be easier

to solve than the original problem Perform classifications fast

Rapid Classification Tree

Example RCT1,..,20

2,7,11,12,14,16,17,19

7,11,16,17

11,177,16

14,192,12

1,13,15,18

13,181,15

5,6,9,103,4,8,20

- Indicates that all children are leaf nodes

Principal Component Analysis Classical technique that is widely used for

image compression and recognition Produces features with a dimensionality

significantly less than that of the original images

Reduction is performed without a substantial loss of the data contained in the image

Analysis is based on the variance of dataset◦ Variance implies a distinction in class

PCA

47-229-498179106-157-3461124-99-257423

FeatureSet

478-367206-358386

LowerDimensional

Space

FeatureSet

PCAMatrix×

In many cases, the PCA reduction was not sufficient

Improving the performance of the reduction matrix is necessary

Four methods were implemented◦ Gradient Search◦ Random or Vibration Search

Variation of the Metropolis Algorithm◦ Neighborhood Component Analysis◦ Stochastic Gradient Search

3. Reduction Optimization

Optimization Search Data reduction occurs via a matrix

multiplication◦ x′ = xA

Optimization is achieved by ◦ defining F as a function A, F(A)◦ Changing A

Gradient Search

Gradient Search Can be computationally expensive Does not provide a means to escape a

local maximum

x

f(x )

lo c a lm a x im u m

g lo b a lm a x im u m

Stochastic Search Makes a guess Guesses are fast There is a possibility of escaping a local

maximum

x

f(x )

lo c a lm a x im u m

g lo b a lm a x im u m

Stochastic Search Restricting the search area increases the

probability of finding an increasing path

If all data at the node can be classified accurately◦ The classification decision is stored as leaf nodes◦ No further processing occurs down this branch of the

tree. If data cannot be classified accurately

◦ The problem then becomes a clustering one.◦ Accuracy is now defined in terms of clusters, not classes.

The accuracy achieved through class level clustering will always be no worse than that of individual classes and in most cases, will be higher.

Clustering

F e a tu re S p a c e D e c is io n B o u n d a ry P rio r to C lu s te r in g

D e c is io n B o u n d a ry A f te r C lu s te r in g

Clustering Process

Decomposition

2

4

105

1

3

796

8

R

AB C

D

E

Generated RCT

F e a ture S pac e D e c is ion B ou nd ary A fte r C lu ster in g

A

R o o t

B

Decomposition2

4

105

1

3

796

8

R

AB C

D

E

Decomposition

F ea tu re S pa ce D e c isio n B ou nd a ry A fte r C lu sterin g

C

D

E

Decomposition2

4

105

1

3

796

8

R

AB C

D

E

Classifier Dataset Accuracy ThroughputKNN 100% 16,735Neural Network 92.50% 53,522SVM 86.25% 7,251RCT 97% 982,110

KNN 100% 19,204Neural Network 89% 53,709SVM 97.75% 3,790RCT 100% 3,781,933

SVM 96.96% 141,240RCT 97.35% 8,949,638

ORL

COIL

Breast Cancer

Classifier Comparisons

X. Liu, A. Srivastava, K. Gallivan, Optimal linear representations of images for object recognition, IEEE Transactions on Pattern Analysis and Machine Intelligence, May 2004, pp. 662-666.

Haynes, K., Liu, X., Mio, W., (2006). Object Recognition Using Rapid Classification Trees. 2006 International Conference on Image Processing

Haynes, K. (2011). Using Image Steganography to Establish Covert Communication Channels. International Journal of Computer Science and Information Security, Vol. 9, No. 9

Duda, R., Hart, P., Stork, D. (2001). Pattern Classification. Wiley-Interscience Publications, NY

References

Questions?