Axel Naumann, DØ University of Nijmegen, The Netherlands June 24, 2002 ACAT02, Moscow 1 Support...

Axel Naumann, DAxel Naumann, DØØUniversity of Nijmegen, The University of Nijmegen, The

NetherlandsNetherlands

June 24, 2002June 24, 2002

ACAT02, MoscowACAT02, Moscow 11

Support Vector Regression

Axel Naumann, DAxel Naumann, DØØUniversity of Nijmegen, The University of Nijmegen, The

NetherlandsNetherlands

June 24, 2002June 24, 2002

ACAT02, MoscowACAT02, Moscow 22

SVR

Drawings and illustrations from Bernhard Schölkopf, and Alex Smola: Learning with Kernels (MIT Press, Cambridge, MA, 2002)

Axel Naumann, DAxel Naumann, DØØUniversity of Nijmegen, The University of Nijmegen, The

NetherlandsNetherlands

June 24, 2002June 24, 2002

ACAT02, MoscowACAT02, Moscow 33

SVR - History

Based on Learning Theory, consisting of few axioms on learning errors

Started in 1960’s, still actively developed

SVRs recently outperformed NNs in recognition tests on US Postal Service’s standard set of handwritten characters

libSVM by Chih-Chung Chang and Chih-Jen Lin provides fast and simple to use implementation, extended as requests (e.g. from HEP) come in

Axel Naumann, DAxel Naumann, DØØUniversity of Nijmegen, The University of Nijmegen, The

NetherlandsNetherlands

June 24, 2002June 24, 2002

ACAT02, MoscowACAT02, Moscow 44

Training sample X, observed results YGoal: f with y=f(x)

Simplicity: • Linear case,•

Formulation of Problem

miyxf

bxwxf

ii ,,1

,

1 ,1y

Axel Naumann, DAxel Naumann, DØØUniversity of Nijmegen, The University of Nijmegen, The

NetherlandsNetherlands

June 24, 2002June 24, 2002

ACAT02, MoscowACAT02, Moscow 55

Optimal confidence = maximal margin

Minimize quadratic problem

with Quadratic problem: Unique solution!

Optimizing the Confidence

m

i iii

m

i ii

m

ji jijjii

y

xx

1

*

1

*

1,

** ,2

1

m

i iii

m

i ii xyw11

* ;0

Axel Naumann, DAxel Naumann, DØØUniversity of Nijmegen, The University of Nijmegen, The

NetherlandsNetherlands

June 24, 2002June 24, 2002

ACAT02, MoscowACAT02, Moscow 66

Non-Linearity

bxxkbxwxf

xxkxxxxm

i ii

, ,

,, ,

1

*

:Introduce mapping to higher dimensional space

e.g. Gaussian kernel:

2

2

2exp,

xx

xxk

Axel Naumann, DAxel Naumann, DØØUniversity of Nijmegen, The University of Nijmegen, The

NetherlandsNetherlands

June 24, 2002June 24, 2002

ACAT02, MoscowACAT02, Moscow 77

Calculation

Axel Naumann, DAxel Naumann, DØØUniversity of Nijmegen, The University of Nijmegen, The

NetherlandsNetherlands

June 24, 2002June 24, 2002

ACAT02, MoscowACAT02, Moscow 88

L2 b Tagger Parameters

Axel Naumann, DAxel Naumann, DØØUniversity of Nijmegen, The University of Nijmegen, The

NetherlandsNetherlands

June 24, 2002June 24, 2002

ACAT02, MoscowACAT02, Moscow 99

L2 b Tagger Parameters

Axel Naumann, DAxel Naumann, DØØUniversity of Nijmegen, The University of Nijmegen, The

NetherlandsNetherlands

June 24, 2002June 24, 2002

ACAT02, MoscowACAT02, Moscow 1010

L2 b Tagger Output

SVR NN

Axel Naumann, DAxel Naumann, DØØUniversity of Nijmegen, The University of Nijmegen, The

NetherlandsNetherlands

June 24, 2002June 24, 2002

ACAT02, MoscowACAT02, Moscow 1111

L2 b Tagger Discussion

• Complex problem increases number of SVs• Almost non-separable classes still almost non-

separable in high dimensional space• High processing time due to large number of

SVs

NNs show better performance for low-information, low-separability problems

Axel Naumann, DAxel Naumann, DØØUniversity of Nijmegen, The University of Nijmegen, The

NetherlandsNetherlands

June 24, 2002June 24, 2002

ACAT02, MoscowACAT02, Moscow 1212

Higgs Parameters

Higgs SVR analysis by Daniel Whiteson, UC Berkley

Axel Naumann, DAxel Naumann, DØØUniversity of Nijmegen, The University of Nijmegen, The

NetherlandsNetherlands

June 24, 2002June 24, 2002

ACAT02, MoscowACAT02, Moscow 1313

Higgs Parameters

Axel Naumann, DAxel Naumann, DØØUniversity of Nijmegen, The University of Nijmegen, The

NetherlandsNetherlands

June 24, 2002June 24, 2002

ACAT02, MoscowACAT02, Moscow 1414

Higgs Output

Background Signal Background Signal

Axel Naumann, DAxel Naumann, DØØUniversity of Nijmegen, The University of Nijmegen, The

NetherlandsNetherlands

June 24, 2002June 24, 2002

ACAT02, MoscowACAT02, Moscow 1515

Higgs Purity / EfficiencyPurity

Axel Naumann, DAxel Naumann, DØØUniversity of Nijmegen, The University of Nijmegen, The

NetherlandsNetherlands

June 24, 2002June 24, 2002

ACAT02, MoscowACAT02, Moscow 1616

Kernel Width

k

kkji

ji

xx

xx

2

22

2/exp

: widthsionalmultidimen

2/exp

:original

Kernel Width

Inte

gra

ted

Sig

nifi

can

ce

Axel Naumann, DAxel Naumann, DØØUniversity of Nijmegen, The University of Nijmegen, The

NetherlandsNetherlands

June 24, 2002June 24, 2002

ACAT02, MoscowACAT02, Moscow 1717

Summary

SVR often superior to NN• Not stuck in local minima: unique solution• Better performance for many problemsImplementation exists, actively supported by the

development community

Further information: www.kernel-machines.org

Time for SVR @ HEP!

Axel Naumann, DAxel Naumann, DØØUniversity of Nijmegen, The University of Nijmegen, The

NetherlandsNetherlands

June 24, 2002June 24, 2002

ACAT02, MoscowACAT02, Moscow 1818

L2 b Tagger Correlation

b udcsSVR

SVR

NN

NN