4/5/2004 JHU-APL 1

Machine Learning Machine Learning forfor Image RetrievalImage Retrieval

Edward ChangAssociate Professor,

Electrical Engineering, UC Santa BarbaraCTO, VIMA Technologies

4/5/2004 JHU-APL 2

Are They Similar?

4/5/2004 JHU-APL 3

Are They Similar?In terms of what?Is Tiger Woods more similar to Michael Jordan than to Bill Gates?

4/5/2004 JHU-APL 4

Conveying In Terms of What

Relational DatabasesConveyed via Query Languages

Example Select * where colors = (blue v pink v white)

& (textures = coarse v horizontal)& (shapes = people + tennis rackets)

4/5/2004 JHU-APL 5

Conveying In Terms of What

Image DatabasesConveyed via Examples

Use a sunset picture (or pictures) to find more sunset imagesWhere does the perfect example come from?

4/5/2004 JHU-APL 6

Conveying In Terms of What

Internet SearchesConveyed via Keywords

4/5/2004 JHU-APL 7

4/5/2004 JHU-APL 8

4/5/2004 JHU-APL 9

4/5/2004 JHU-APL 10

Query by KeywordsPros

A user-friendly paradigmCons

Annotation is a laborious processAnnotation quality can be subparAnnotation can be subjectiveSynonyms

4/5/2004 JHU-APL 11

Query Specification Paradigms

Query by SQL-like languagesQuery by examplesQuery by keywords

Query by nothing !

4/5/2004 JHU-APL 12

Image Retrieval Demo

ACM SIGMOD 01 ACM MM 01, 02IEEE CVPR 03NSF Paris, Harvard, DC, Seattle Workshops

4/5/2004 JHU-APL 13

Outline

Query ParadigmsQ-by-Nothing DemoTechnical ChallengesPreliminary ResultsTechnology Summary

4/5/2004 JHU-APL 14

Technical Challenges

Learn a complex and subjective query conceptFormulate a distance function to measure perceptual similarity

4/5/2004 JHU-APL 15

Classical Statistical Models [Donoho 2000]

N:Number of training instancesD:DimensionalityClassical models assume

N >> D N → ∞N- ≈ N+

4/5/2004 JHU-APL 16

Image (Information) Retrieval

N < DN+ << N-

4/5/2004 JHU-APL 17

SolutionsN < D

Make each u in U most informative ⌧ACM TIOS 2003, ACM MM 2001

Increase N- through co-training ⌧PCM 2002, ICIP 2003

Reduce D⌧ACM MM 2002 (DPF)

N+ << N-

Conformal transformation Kernel boundary alignment⌧ACM MM 2003, ICML 2003

Advanced Methods

4/5/2004 JHU-APL 18

Query Concept

4/5/2004 JHU-APL 19

Traditional Random Sampling

4/5/2004 JHU-APL 20

MEGA’s S- & F-Step

4/5/2004 JHU-APL 21

SVMActive

4/5/2004 JHU-APL 22

SVMActive

4/5/2004 JHU-APL 23

SVMActive

4/5/2004 JHU-APL 24

SVMActive

4/5/2004 JHU-APL 25

Ranking

4/5/2004 JHU-APL 26

SolutionsN < D

Make each u in U most informative ⌧ACM TIOS 2003, ACM MM 2001

Increase N- through co-training ⌧PCM 2002, ICIP 2003

Reduce D⌧ACM MM 2002 (DPF)

N+ << N-

Conformal transformation Kernel boundary alignment⌧ACM MM 2003, ICML 2003

Advanced Methods

4/5/2004 JHU-APL 27

A Checkerboard Experiment

MinoritiesMajorities

Ratio Imbalanced

=

+ :: Majorityo :: Minority

10:1

4/5/2004 JHU-APL 28

A Checkerboard Experiment

MinoritiesMajorities

Ratio Imbalanced

=

ideal

+ :: Majorityo :: Minority

10:1

4/5/2004 JHU-APL 29

A Checkerboard Experiment

MinoritiesMajorities

Ratio Imbalanced

=

+ :: Majorityo :: Minority

10:1

4/5/2004 JHU-APL 30

Boundary Bias in SVMs

Majority

Minority

Majority

Minority

4/5/2004 JHU-APL 31

Bayesian Explanation

Bayes Decision Rule

When N- >> N+, p(ω−) >> p(ω+)

( ) ( ) ( ))(

| |x

xxpppp −−

− =ωωω( ) ( ) ( )

)(| |x

xxpppp ++

+ =ωωω

( )( ) ωω

ωω class ofprior :

class of likelihood :|pp x( )

( )( )( )+

−

−

+ ≥ωω

ωω

pp

pp

||xx

4/5/2004 JHU-APL 32

Three Strategies [ICML 2003]

+=∑

=

bKfn

iiii

1),(y)(sgn xxx α

b : interceptb : influence of xii

K : kernel functionα

4/5/2004 JHU-APL 33

Imbalanced Data SetImbalanced Data Set

ideal

trained

Majority

Minority

SV-

SV+

4/5/2004 JHU-APL 34

After AlignmentAfter Alignment

idealKBA trained

Majority

Minority

4/5/2004 JHU-APL 35

Technical Challenges

Learn a complex and subjective query conceptFormulate a distance function to measure similarity

4/5/2004 JHU-APL 36

Are They Similar?

4/5/2004 JHU-APL 37

Perceptual Distance FunctionTwo Monumental Challenges

Formulating a perceptual feature spaceFormulating a perceptual distance function

4/5/2004 JHU-APL 38

Distribution of Distances

4/5/2004 JHU-APL 39

Minkowski Distance

Objects P and QD = (ΣM (pi - qi)n)1/n

Similar images are similar in all M features

4/5/2004 JHU-APL 40

1.0E-06

1.0E-05

1.0E-04

1.0E-03

1.0E-02

1.0E-01

00.

060.

130.

190.

250.

320.

380.

440.

510.

570.

630.

690.

760.

820.

880.

95

Feature Distance

Freq

uenc

y

1.0E-06

1.0E-05

1.0E-04

1.0E-03

1.0E-02

1.0E-01

00.

060.

130.

19

0.25

0.32

0.38

0.44

0.51

0.57

0.63

0.69

0.76

0.82

0.88

0.95

Feature Distance

Freq

uenc

y

4/5/2004 JHU-APL 41

Weighted Minkowski Distance

D = (ΣM wi(pi - qi)n)1/n

Similar images are similar in the same subset of the M features

4/5/2004 JHU-APL 42

0 0 0 0 0 0 00 0 0 0 0 0 00 0 0 0 0 0 00 0 0 0 0 0 00 0 0 0 0 0 00 0 0 0 0 0 00 0 0 0 0 0 00 0 0 0 0 0 00 0 0 0 0 0 0

0.007545 0.01307 0.004637 0.002413 0.002635 0.002954 0.0020070.014669 0.02717 0.010578 0.006734 0.007725 0.006379 0.0057660.012615 0.023055 0.009333 0.006764 0.007363 0.006593 0.0054430.082128 0.212612 0.068016 0.037835 0.032241 0.018068 0.0132030.061564 0.176548 0.045542 0.026445 0.026374 0.018583 0.0220370.019243 0.037016 0.015684 0.010834 0.012792 0.013536 0.0093460.09418 0.153677 0.066896 0.040249 0.036368 0.030341 0.0211380.1284 0.335405 0.13774 0.072613 0.054947 0.039216 0.043319

0.041414 0.101403 0.035881 0.022633 0.018991 0.017131 0.019450.014024 0.049782 0.01457 0.0053 0.004439 0.003041 0.0052260.049319 0.120274 0.045804 0.020165 0.019499 0.013805 0.018513

GIF

00.020.040.060.080.1

0.120.14

1 11 21 31 41 51 61 71 81 91 101

111

121

131

141

Feature Number

Ave

rage

Dis

tanc

e0 0 0 0 0 0 0 00 0 0 0 0 0 0 00 0 0 0 0 0 0 00 0 0 0 0 0 0 00 0 0 0 0 0 0 00 0 0 0 0 0 0 00 0 0 0 0 0 0 00 0 0 0 0 0 0 00 0 0 0 0 0 0 0

0.002923 0.004377 0.029086 0.017063 0.007649 0.002019 0.001984 0.011560.006648 0.010143 0.070708 0.046142 0.023502 0.005178 0.005169 0.030140.006298 0.009264 0.075118 0.042225 0.020053 0.006285 0.006533 0.0300430.010198 0.056025 0.052869 0.033199 0.018294 0.00688 0.006858 0.023620.017066 0.047514 0.104013 0.073459 0.037468 0.013849 0.01293 0.0483440.008148 0.015337 0.074134 0.044238 0.021222 0.005197 0.005099 0.0299780.013529 0.051743 0.063263 0.038084 0.020885 0.010481 0.009844 0.0285110.045746 0.104141 0.145924 0.11276 0.065015 0.026333 0.02593 0.0751920.026167 0.034522 0.085067 0.054154 0.02918 0.015887 0.014371 0.0397320.002676 0.012148 0.008913 0.004682 0.002452 0.000913 0.000905 0.0035730.014527 0.036084 0.046779 0.024712 0.017418 0.004182 0.004991 0.0196160.012121 0.030269 0.045198 0.022268 0.012468 0.004706 0.004955 0.017919

Scale up/down

00.050.1

0.150.2

0.250.3

0.350.4

1 11 21 31 41 51 61 71 81 91 101

111

121

131

141

Feature Number

Aver

age

Dis

tanc

e

0.024788 0.069615 0.0226 0.009364 0.01 0.00678 0.0097120.094781 0.227558 0.099002 0.046466 0.047815 0.036883 0.0246990.093399 0.233519 0.188091 0.043026 0.037991 0.022151 0.0240640.040228 0.102763 0.034949 0.014184 0.01465 0.010237 0.0155170.001163 0.000896 0.000722 0.000627 0.000349 0.000452 0.0027580.006947 0.006769 0.003541 0.006377 0.002048 0.005515 0.0130060.006365 0.005313 0.002064 0.004006 0.002055 0.003338 0.01010.011705 0.010935 0.006615 0.007506 0.003319 0.005911 0.0152110.009434 0.010169 0.004484 0.006306 0.002582 0.004798 0.0136570.006305 0.005997 0.003392 0.005719 0.002382 0.004853 0.0128020.005835 0.00945 0.004323 0.00564 0.002688 0.004535 0.0063320.008149 0.009636 0.0047 0.006213 0.002564 0.003375 0.0064210.006776 0.010315 0.005393 0.008004 0.003845 0.005659 0.0132030.001526 0.002551 0.000576 0.000371 0.000331 0.000286 0.000380.016302 0.022657 0.007055 0.00353 0.002171 0.004162 0.003980.012414 0.020159 0.007076 0.003102 0.00188 0.004606 0.003490.007231 0.013591 0.004979 0.001092 0.000582 0.002766 0.0007410.011588 0.015102 0.005764 0.003855 0.00262 0.004584 0.0037920.01212 0.016013 0.006441 0.004048 0.002728 0.004856 0.004241

0.012235 0.01671 0.00483 0.002616 0.00197 0.00268 0.001672

Cropping

00.050.1

0.150.2

0.250.3

0.35

1 11 21 31 41 51 61 71 81 91 101

111

121

131

141

Feature Number

Ave

rage

Dis

tanc

e

0.006109 0.019169 0.032795 0.015229 0.008667 0.002357 0.00292 0.0123940.01223 0.070665 0.046472 0.02549 0.017445 0.008694 0.00841 0.021302

0.019067 0.08113 0.04592 0.024327 0.014169 0.004995 0.005275 0.0189370.011323 0.029089 0.063856 0.037716 0.01988 0.00522 0.005556 0.0264460.000995 0.000971 0.00241 0.001415 0.000736 0.000275 0.000272 0.0010220.007103 0.006337 0.015615 0.008709 0.003433 0.001572 0.002071 0.006280.004321 0.004457 0.012494 0.007507 0.003403 0.001351 0.001976 0.0053460.007451 0.008135 0.017145 0.008711 0.003192 0.001154 0.00223 0.0064860.00576 0.006822 0.015235 0.00869 0.003676 0.001193 0.002159 0.006191

0.006491 0.005948 0.013473 0.007436 0.003165 0.001777 0.002377 0.0056460.003832 0.005257 0.011884 0.008077 0.002654 0.001227 0.001213 0.0050110.004812 0.005389 0.011737 0.00729 0.003216 0.001534 0.002039 0.0051630.008795 0.007888 0.016303 0.008801 0.004048 0.002367 0.0027 0.0068440.000451 0.000707 0.002277 0.001346 0.000797 0.000253 0.000239 0.0009820.004914 0.006924 0.01499 0.009123 0.006657 0.003364 0.003391 0.0075050.004473 0.006398 0.017247 0.008858 0.005219 0.002338 0.002392 0.0072110.001723 0.003639 0.010426 0.005216 0.003024 0.00043 0.000423 0.0039040.00427 0.005712 0.011221 0.00856 0.006923 0.004464 0.004462 0.007126

0.004978 0.006186 0.009864 0.007161 0.005881 0.003835 0.003847 0.0061180.001722 0.0046 0.015611 0.007291 0.00338 0.000508 0.00049 0.005456

Rotation

0

0.02

0.04

0.06

0.08

0.1

0.12

1 10 19 28 37 46 55 64 73 82 91 100

109

118

127

136

Feature Number

Ave

rage

Dis

tanc

e

4/5/2004 JHU-APL 43

Similarity Theories

Objects are similar in all respects (Richardson 1928)Objects are similar in some respects (Tversky 1977)Similarity is a process of determining respects, rather than using predefined respects (Goldstone 94)

4/5/2004 JHU-APL 44

DPF

Which Place is Similar to New York?PartialDynamicDynamic Partial Function

4/5/2004 JHU-APL 45

Precision/Recall

4/5/2004 JHU-APL 46

Search: Image Piracy Detection

Unlicensed

“Corbis filed a multi-million dollar lawsuit against Amazon.com, accusing the e-commerce giant of the selling its images without its consent.” –InternetNews.com, July 1, 2003

“The Corbis suit …is seeking … up to $150,000 for each copyrighted work infringed…”

4/5/2004 JHU-APL 47

4/5/2004 JHU-APL 48

4/5/2004 JHU-APL 49

VIMA Visual Search

4/5/2004 JHU-APL 50

Current Other Work

Video Surveillance and Sensor NetworksContext-based Distance Function LearningHigh-dimensional IndexingScalabilitySpeeding up SVMs

4/5/2004 JHU-APL 51

IR Key Components

Learners

Multimodal

IntegrationHigh D

IndexersPerceptual

Distance Function