AdaBoost

Lecturer:Jan Sochman

Authors:Jan Sochman, Jirı Matas

Centre for Machine PerceptionCzech Technical University, Prague

http://cmp.felk.cvut.cz

2/12

Presentation

Outline:

� AdaBoost algorithm

• Why is of interest?

• How it works?

• Why it works?

� AdaBoost variants

3/12

What is AdaBoost?

AdaBoost is an algorithm for constructing a “strong” classifier as linear combination

f(x) =T∑

t=1

αtht(x)

of “simple” “weak” classifiers ht(x).

Terminology

� ht(x) ... “weak” or basis classifier, hypothesis, “feature”

� H(x) = sign(f(x)) ... ‘’strong” or final classifier/hypothesis

Interesting properties

� AB is a linear classifier with all its desirable properties.

� AB output converges to the logarithm of likelihood ratio.

� AB has good generalisation properties.

� AB is a feature selector with a principled strategy (minimisation of upper bound onempirical error).

� AB close to sequential decision making (it produces a sequence of gradually morecomplex classifiers).

4/12

The AdaBoost Algorithm

Given: (x1, y1), . . . , (xm, ym);xi ∈ X , yi ∈ {−1,+1}

4/12

The AdaBoost Algorithm

Given: (x1, y1), . . . , (xm, ym);xi ∈ X , yi ∈ {−1,+1}Initialise weights D1(i) = 1/m.

4/12

The AdaBoost Algorithm

Given: (x1, y1), . . . , (xm, ym);xi ∈ X , yi ∈ {−1,+1}Initialise weights D1(i) = 1/m.For t = 1, ..., T :

4/12

The AdaBoost Algorithm

Given: (x1, y1), . . . , (xm, ym);xi ∈ X , yi ∈ {−1,+1}Initialise weights D1(i) = 1/m.For t = 1, ..., T :

� Find ht = arg minhj∈H

εj =m∑

i=1

Dt(i)Jyi 6= hj(xi)K

t = 1

4/12

The AdaBoost Algorithm

Given: (x1, y1), . . . , (xm, ym);xi ∈ X , yi ∈ {−1,+1}Initialise weights D1(i) = 1/m.For t = 1, ..., T :

� Find ht = arg minhj∈H

εj =m∑

i=1

Dt(i)Jyi 6= hj(xi)K

� If εt ≥ 1/2 then stop

t = 1

4/12

The AdaBoost Algorithm

Given: (x1, y1), . . . , (xm, ym);xi ∈ X , yi ∈ {−1,+1}Initialise weights D1(i) = 1/m.For t = 1, ..., T :

� Find ht = arg minhj∈H

εj =m∑

i=1

Dt(i)Jyi 6= hj(xi)K

� If εt ≥ 1/2 then stop

� Set αt = 12 log(1−εt

εt)

t = 1

4/12

The AdaBoost Algorithm

Given: (x1, y1), . . . , (xm, ym);xi ∈ X , yi ∈ {−1,+1}Initialise weights D1(i) = 1/m.For t = 1, ..., T :

� Find ht = arg minhj∈H

εj =m∑

i=1

Dt(i)Jyi 6= hj(xi)K

� If εt ≥ 1/2 then stop

� Set αt = 12 log(1−εt

εt)

� Update

Dt+1(i) =Dt(i)exp(−αtyiht(xi))

Zt

t = 1

4/12

The AdaBoost Algorithm

Given: (x1, y1), . . . , (xm, ym);xi ∈ X , yi ∈ {−1,+1}Initialise weights D1(i) = 1/m.For t = 1, ..., T :

� Find ht = arg minhj∈H

εj =m∑

i=1

Dt(i)Jyi 6= hj(xi)K

� If εt ≥ 1/2 then stop

� Set αt = 12 log(1−εt

εt)

� Update

Dt+1(i) =Dt(i)exp(−αtyiht(xi))

Zt

Output the final classifier:

H(x) = sign

(T∑

t=1

αtht(x)

)

t = 1

0 5 10 15 20 25 30 35 400

0.05

0.1

0.15

0.2

0.25

0.3

0.35

4/12

The AdaBoost Algorithm

Given: (x1, y1), . . . , (xm, ym);xi ∈ X , yi ∈ {−1,+1}Initialise weights D1(i) = 1/m.For t = 1, ..., T :

� Find ht = arg minhj∈H

εj =m∑

i=1

Dt(i)Jyi 6= hj(xi)K

� If εt ≥ 1/2 then stop

� Set αt = 12 log(1−εt

εt)

� Update

Dt+1(i) =Dt(i)exp(−αtyiht(xi))

Zt

Output the final classifier:

H(x) = sign

(T∑

t=1

αtht(x)

)

t = 2

0 5 10 15 20 25 30 35 400

0.05

0.1

0.15

0.2

0.25

0.3

0.35

4/12

The AdaBoost Algorithm

Given: (x1, y1), . . . , (xm, ym);xi ∈ X , yi ∈ {−1,+1}Initialise weights D1(i) = 1/m.For t = 1, ..., T :

� Find ht = arg minhj∈H

εj =m∑

i=1

Dt(i)Jyi 6= hj(xi)K

� If εt ≥ 1/2 then stop

� Set αt = 12 log(1−εt

εt)

� Update

Dt+1(i) =Dt(i)exp(−αtyiht(xi))

Zt

Output the final classifier:

H(x) = sign

(T∑

t=1

αtht(x)

)

t = 3

0 5 10 15 20 25 30 35 400

0.05

0.1

0.15

0.2

0.25

0.3

0.35

4/12

The AdaBoost Algorithm

Given: (x1, y1), . . . , (xm, ym);xi ∈ X , yi ∈ {−1,+1}Initialise weights D1(i) = 1/m.For t = 1, ..., T :

� Find ht = arg minhj∈H

εj =m∑

i=1

Dt(i)Jyi 6= hj(xi)K

� If εt ≥ 1/2 then stop

� Set αt = 12 log(1−εt

εt)

� Update

Dt+1(i) =Dt(i)exp(−αtyiht(xi))

Zt

Output the final classifier:

H(x) = sign

(T∑

t=1

αtht(x)

)

t = 4

0 5 10 15 20 25 30 35 400

0.05

0.1

0.15

0.2

0.25

0.3

0.35

4/12

The AdaBoost Algorithm

Given: (x1, y1), . . . , (xm, ym);xi ∈ X , yi ∈ {−1,+1}Initialise weights D1(i) = 1/m.For t = 1, ..., T :

� Find ht = arg minhj∈H

εj =m∑

i=1

Dt(i)Jyi 6= hj(xi)K

� If εt ≥ 1/2 then stop

� Set αt = 12 log(1−εt

εt)

� Update

Dt+1(i) =Dt(i)exp(−αtyiht(xi))

Zt

Output the final classifier:

H(x) = sign

(T∑

t=1

αtht(x)

)

t = 5

0 5 10 15 20 25 30 35 400

0.05

0.1

0.15

0.2

0.25

0.3

0.35

4/12

The AdaBoost Algorithm

Given: (x1, y1), . . . , (xm, ym);xi ∈ X , yi ∈ {−1,+1}Initialise weights D1(i) = 1/m.For t = 1, ..., T :

� Find ht = arg minhj∈H

εj =m∑

i=1

Dt(i)Jyi 6= hj(xi)K

� If εt ≥ 1/2 then stop

� Set αt = 12 log(1−εt

εt)

� Update

Dt+1(i) =Dt(i)exp(−αtyiht(xi))

Zt

Output the final classifier:

H(x) = sign

(T∑

t=1

αtht(x)

)

t = 6

0 5 10 15 20 25 30 35 400

0.05

0.1

0.15

0.2

0.25

0.3

0.35

4/12

The AdaBoost Algorithm

Given: (x1, y1), . . . , (xm, ym);xi ∈ X , yi ∈ {−1,+1}Initialise weights D1(i) = 1/m.For t = 1, ..., T :

� Find ht = arg minhj∈H

εj =m∑

i=1

Dt(i)Jyi 6= hj(xi)K

� If εt ≥ 1/2 then stop

� Set αt = 12 log(1−εt

εt)

� Update

Dt+1(i) =Dt(i)exp(−αtyiht(xi))

Zt

Output the final classifier:

H(x) = sign

(T∑

t=1

αtht(x)

)

t = 7

0 5 10 15 20 25 30 35 400

0.05

0.1

0.15

0.2

0.25

0.3

0.35

4/12

The AdaBoost Algorithm

Given: (x1, y1), . . . , (xm, ym);xi ∈ X , yi ∈ {−1,+1}Initialise weights D1(i) = 1/m.For t = 1, ..., T :

� Find ht = arg minhj∈H

εj =m∑

i=1

Dt(i)Jyi 6= hj(xi)K

� If εt ≥ 1/2 then stop

� Set αt = 12 log(1−εt

εt)

� Update

Dt+1(i) =Dt(i)exp(−αtyiht(xi))

Zt

Output the final classifier:

H(x) = sign

(T∑

t=1

αtht(x)

)

t = 40

0 5 10 15 20 25 30 35 400

0.05

0.1

0.15

0.2

0.25

0.3

0.35

5/12

Reweighting

Effect on the training set

Dt+1(i) =Dt(i)exp(−αtyiht(xi))

Zt

exp(−αtyiht(xi)){

< 1, yi = ht(xi)> 1, yi 6= ht(xi)

−2 −1.5 −1 −0.5 0 0.5 1 1.5 2

0

0.5

1

1.5

2

2.5

3

yf(x)

err

⇒ Increase (decrease) weight of wrongly (correctly) classified examples

⇒ The weight is the upper bound on the error of a given example.

⇒ All information about previously selected “features” is captured in Dt.

5/12

Reweighting

Effect on the training set

Dt+1(i) =Dt(i)exp(−αtyiht(xi))

Zt

exp(−αtyiht(xi)){

< 1, yi = ht(xi)> 1, yi 6= ht(xi)

−2 −1.5 −1 −0.5 0 0.5 1 1.5 2

0

0.5

1

1.5

2

2.5

3

yf(x)

err

⇒ Increase (decrease) weight of wrongly (correctly) classified examples

⇒ The weight is the upper bound on the error of a given example.

⇒ All information about previously selected “features” is captured in Dt.

5/12

Reweighting

Effect on the training set

Dt+1(i) =Dt(i)exp(−αtyiht(xi))

Zt

exp(−αtyiht(xi)){

< 1, yi = ht(xi)> 1, yi 6= ht(xi)

−2 −1.5 −1 −0.5 0 0.5 1 1.5 2

0

0.5

1

1.5

2

2.5

3

yf(x)

err

⇒ Increase (decrease) weight of wrongly (correctly) classified examples

⇒ The weight is the upper bound on the error of a given example.

⇒ All information about previously selected “features” is captured in Dt.

5/12

Reweighting

Effect on the training set

Dt+1(i) =Dt(i)exp(−αtyiht(xi))

Zt

exp(−αtyiht(xi)){

< 1, yi = ht(xi)> 1, yi 6= ht(xi)

−2 −1.5 −1 −0.5 0 0.5 1 1.5 2

0

0.5

1

1.5

2

2.5

3

yf(x)

err

⇒ Increase (decrease) weight of wrongly (correctly) classified examples

⇒ The weight is the upper bound on the error of a given example.

⇒ All information about previously selected “features” is captured in Dt.

6/12

Upper Bound Theorem

Theorem: The following upper bound holds on the training error of H

1m|{i : H(xi) 6= yi}| ≤

T∏t=1

Zt

Proof: By unravelling the update rule

DT+1(i) =exp(−

∑t αtyiht(xi))

m∏

t Zt

=exp(−yif(xi))

m∏

t Zt.

If H(xi) 6= yi then yif(xi) ≤ 0 implying that exp(−yif(xi)) > 1, thus

JH(xi) 6= yiK ≤ exp(−yif(xi))1m

∑i

JH(xi) 6= yiK ≤ 1m

∑i

exp(−yif(xi))

=∑

i

(∏

t

Zt)DT+1(i) =∏

t

Zt−2 −1.5 −1 −0.5 0 0.5 1 1.5 2

0

0.5

1

1.5

2

2.5

3

yf(x)

err

7/12

Consequences of the Theorem

� Instead of minimising the training error, its upper bound can be minimised.

� This can be done by minimising Zt in each training round by:

• Choosing optimal ht, and

• Finding optimal αt.

� Can be proved to maximise margin.

8/12

Choosing αt

We attempt to minimise Zt =∑

i Dt(i)exp(−αtyiht(xi)):

dZ

dα= −

m∑i=1

D(i)yih(xi)e−yiαih(xi) = 0

−∑

i:yi=h(xi)

D(i)e−α +∑

i:yi 6=h(xi)

D(i)eα = 0

−e−α(1− ε) + eαε = 0

α =12

log1− ε

ε

⇒ The minimisator of the upper bound is αt = 12 log 1−εt

εt.

9/12

Choosing ht

Weak classifier examples

� Decision tree, Perceptron – H infinite

� Selecting the best one from given finite set H

Justification of the weighted error minimisation

Having αt = 12 log 1−εt

εt

Zt =m∑

i=1

Dt(i)e−yiαiht(xi)

=∑

i:yi=ht(xi)

Dt(i)e−αt +∑

i:yi 6=ht(xi)

Dt(i)eαt

= (1− εt)e−αt + εteαt

= 2√

εt(1− εt)

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

εt

Z t

⇒ Zt is minimised by selecting ht with minimal weighted error εt.

10/12

The Algorithm Recapitulation

Given: (x1, y1), . . . , (xm, ym);xi ∈ X , yi ∈ {−1,+1}

10/12

The Algorithm Recapitulation

Given: (x1, y1), . . . , (xm, ym);xi ∈ X , yi ∈ {−1,+1}Initialise weights D1(i) = 1/m.

10/12

The Algorithm Recapitulation

Given: (x1, y1), . . . , (xm, ym);xi ∈ X , yi ∈ {−1,+1}Initialise weights D1(i) = 1/m.For t = 1, ..., T :

10/12

The Algorithm Recapitulation

Given: (x1, y1), . . . , (xm, ym);xi ∈ X , yi ∈ {−1,+1}Initialise weights D1(i) = 1/m.For t = 1, ..., T :

� Find ht = arg minhj∈H

εj =m∑

i=1

Dt(i)Jyi 6= hj(xi)K

t = 1

10/12

The Algorithm Recapitulation

Given: (x1, y1), . . . , (xm, ym);xi ∈ X , yi ∈ {−1,+1}Initialise weights D1(i) = 1/m.For t = 1, ..., T :

� Find ht = arg minhj∈H

εj =m∑

i=1

Dt(i)Jyi 6= hj(xi)K

� If εt ≥ 1/2 then stop

t = 1

10/12

The Algorithm Recapitulation

Given: (x1, y1), . . . , (xm, ym);xi ∈ X , yi ∈ {−1,+1}Initialise weights D1(i) = 1/m.For t = 1, ..., T :

� Find ht = arg minhj∈H

εj =m∑

i=1

Dt(i)Jyi 6= hj(xi)K

� If εt ≥ 1/2 then stop

� Set αt = 12 log(1−εt

εt)

t = 1

10/12

The Algorithm Recapitulation

Given: (x1, y1), . . . , (xm, ym);xi ∈ X , yi ∈ {−1,+1}Initialise weights D1(i) = 1/m.For t = 1, ..., T :

� Find ht = arg minhj∈H

εj =m∑

i=1

Dt(i)Jyi 6= hj(xi)K

� If εt ≥ 1/2 then stop

� Set αt = 12 log(1−εt

εt)

� Update

Dt+1(i) =Dt(i)exp(−αtyiht(xi))

Zt

t = 1

10/12

The Algorithm Recapitulation

Given: (x1, y1), . . . , (xm, ym);xi ∈ X , yi ∈ {−1,+1}Initialise weights D1(i) = 1/m.For t = 1, ..., T :

� Find ht = arg minhj∈H

εj =m∑

i=1

Dt(i)Jyi 6= hj(xi)K

� If εt ≥ 1/2 then stop

� Set αt = 12 log(1−εt

εt)

� Update

Dt+1(i) =Dt(i)exp(−αtyiht(xi))

Zt

Output the final classifier:

H(x) = sign

(T∑

t=1

αtht(x)

)

t = 1

0 5 10 15 20 25 30 35 400

0.05

0.1

0.15

0.2

0.25

0.3

0.35

10/12

The Algorithm Recapitulation

Given: (x1, y1), . . . , (xm, ym);xi ∈ X , yi ∈ {−1,+1}Initialise weights D1(i) = 1/m.For t = 1, ..., T :

� Find ht = arg minhj∈H

εj =m∑

i=1

Dt(i)Jyi 6= hj(xi)K

� If εt ≥ 1/2 then stop

� Set αt = 12 log(1−εt

εt)

� Update

Dt+1(i) =Dt(i)exp(−αtyiht(xi))

Zt

Output the final classifier:

H(x) = sign

(T∑

t=1

αtht(x)

)

t = 2

0 5 10 15 20 25 30 35 400

0.05

0.1

0.15

0.2

0.25

0.3

0.35

10/12

The Algorithm Recapitulation

Given: (x1, y1), . . . , (xm, ym);xi ∈ X , yi ∈ {−1,+1}Initialise weights D1(i) = 1/m.For t = 1, ..., T :

� Find ht = arg minhj∈H

εj =m∑

i=1

Dt(i)Jyi 6= hj(xi)K

� If εt ≥ 1/2 then stop

� Set αt = 12 log(1−εt

εt)

� Update

Dt+1(i) =Dt(i)exp(−αtyiht(xi))

Zt

Output the final classifier:

H(x) = sign

(T∑

t=1

αtht(x)

)

t = 3

0 5 10 15 20 25 30 35 400

0.05

0.1

0.15

0.2

0.25

0.3

0.35

10/12

The Algorithm Recapitulation

Given: (x1, y1), . . . , (xm, ym);xi ∈ X , yi ∈ {−1,+1}Initialise weights D1(i) = 1/m.For t = 1, ..., T :

� Find ht = arg minhj∈H

εj =m∑

i=1

Dt(i)Jyi 6= hj(xi)K

� If εt ≥ 1/2 then stop

� Set αt = 12 log(1−εt

εt)

� Update

Dt+1(i) =Dt(i)exp(−αtyiht(xi))

Zt

Output the final classifier:

H(x) = sign

(T∑

t=1

αtht(x)

)

t = 4

0 5 10 15 20 25 30 35 400

0.05

0.1

0.15

0.2

0.25

0.3

0.35

10/12

The Algorithm Recapitulation

Given: (x1, y1), . . . , (xm, ym);xi ∈ X , yi ∈ {−1,+1}Initialise weights D1(i) = 1/m.For t = 1, ..., T :

� Find ht = arg minhj∈H

εj =m∑

i=1

Dt(i)Jyi 6= hj(xi)K

� If εt ≥ 1/2 then stop

� Set αt = 12 log(1−εt

εt)

� Update

Dt+1(i) =Dt(i)exp(−αtyiht(xi))

Zt

Output the final classifier:

H(x) = sign

(T∑

t=1

αtht(x)

)

t = 5

0 5 10 15 20 25 30 35 400

0.05

0.1

0.15

0.2

0.25

0.3

0.35

10/12

The Algorithm Recapitulation

Given: (x1, y1), . . . , (xm, ym);xi ∈ X , yi ∈ {−1,+1}Initialise weights D1(i) = 1/m.For t = 1, ..., T :

� Find ht = arg minhj∈H

εj =m∑

i=1

Dt(i)Jyi 6= hj(xi)K

� If εt ≥ 1/2 then stop

� Set αt = 12 log(1−εt

εt)

� Update

Dt+1(i) =Dt(i)exp(−αtyiht(xi))

Zt

Output the final classifier:

H(x) = sign

(T∑

t=1

αtht(x)

)

t = 6

0 5 10 15 20 25 30 35 400

0.05

0.1

0.15

0.2

0.25

0.3

0.35

10/12

The Algorithm Recapitulation

Given: (x1, y1), . . . , (xm, ym);xi ∈ X , yi ∈ {−1,+1}Initialise weights D1(i) = 1/m.For t = 1, ..., T :

� Find ht = arg minhj∈H

εj =m∑

i=1

Dt(i)Jyi 6= hj(xi)K

� If εt ≥ 1/2 then stop

� Set αt = 12 log(1−εt

εt)

� Update

Dt+1(i) =Dt(i)exp(−αtyiht(xi))

Zt

Output the final classifier:

H(x) = sign

(T∑

t=1

αtht(x)

)

t = 7

0 5 10 15 20 25 30 35 400

0.05

0.1

0.15

0.2

0.25

0.3

0.35

10/12

The Algorithm Recapitulation

Given: (x1, y1), . . . , (xm, ym);xi ∈ X , yi ∈ {−1,+1}Initialise weights D1(i) = 1/m.For t = 1, ..., T :

� Find ht = arg minhj∈H

εj =m∑

i=1

Dt(i)Jyi 6= hj(xi)K

� If εt ≥ 1/2 then stop

� Set αt = 12 log(1−εt

εt)

� Update

Dt+1(i) =Dt(i)exp(−αtyiht(xi))

Zt

Output the final classifier:

H(x) = sign

(T∑

t=1

αtht(x)

)

t = 40

0 5 10 15 20 25 30 35 400

0.05

0.1

0.15

0.2

0.25

0.3

0.35

11/12

AdaBoost variants

Freund & Schapire 1995

� Discrete (h : X → {0, 1})

� Multiclass AdaBoost.M1 (h : X → {0, 1, ..., k})

� Multiclass AdaBoost.M2 (h : X → [0, 1]k)

� Real valued AdaBoost.R (Y = [0, 1], h : X → [0, 1])

Schapire & Singer 1997

� Confidence rated prediction (h : X → R, two-class)

� Multilabel AdaBoost.MR, AdaBoost.MH (different formulation of minimised loss)

... Many other modifications since then (WaldBoost, cascaded AB, online AB, ...)

12/12

Pros and cons of AdaBoost

Advantages

� Very simple to implement

� General learning scheme - can be used for various learning tasks

� Feature selection on very large sets of features

� Fairly good generalisation

Disadvantages

� Suboptimal solution (greedy learning)

� Can overfit in presence of noise (has been addressed recently)

0 5 10 15 20 25 30 35 400

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0 5 10 15 20 25 30 35 400

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0 5 10 15 20 25 30 35 400

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0 5 10 15 20 25 30 35 400

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0 5 10 15 20 25 30 35 400

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0 5 10 15 20 25 30 35 400

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0 5 10 15 20 25 30 35 400

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0 5 10 15 20 25 30 35 400

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0 5 10 15 20 25 30 35 400

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0 5 10 15 20 25 30 35 400

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0 5 10 15 20 25 30 35 400

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0 5 10 15 20 25 30 35 400

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0 5 10 15 20 25 30 35 400

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0 5 10 15 20 25 30 35 400

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0 5 10 15 20 25 30 35 400

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0 5 10 15 20 25 30 35 400

0.05

0.1

0.15

0.2

0.25

0.3

0.35

−2 −1.5 −1 −0.5 0 0.5 1 1.5 2

0

0.5

1

1.5

2

2.5

3

yf(x)

err

−2 −1.5 −1 −0.5 0 0.5 1 1.5 2

0

0.5

1

1.5

2

2.5

3

yf(x)

err

−2 −1.5 −1 −0.5 0 0.5 1 1.5 2

0

0.5

1

1.5

2

2.5

3

yf(x)

err

−2 −1.5 −1 −0.5 0 0.5 1 1.5 2

0

0.5

1

1.5

2

2.5

3

yf(x)

err

−2 −1.5 −1 −0.5 0 0.5 1 1.5 2

0

0.5

1

1.5

2

2.5

3

yf(x)

err

−2 −1.5 −1 −0.5 0 0.5 1 1.5 2

0

0.5

1

1.5

2

2.5

3

yf(x)

err

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

εt

Z t

0 5 10 15 20 25 30 35 400

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0 5 10 15 20 25 30 35 400

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0 5 10 15 20 25 30 35 400

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0 5 10 15 20 25 30 35 400

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0 5 10 15 20 25 30 35 400

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0 5 10 15 20 25 30 35 400

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0 5 10 15 20 25 30 35 400

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0 5 10 15 20 25 30 35 400

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0 5 10 15 20 25 30 35 400

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0 5 10 15 20 25 30 35 400

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0 5 10 15 20 25 30 35 400

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0 5 10 15 20 25 30 35 400

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0 5 10 15 20 25 30 35 400

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0 5 10 15 20 25 30 35 400

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0 5 10 15 20 25 30 35 400

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0 5 10 15 20 25 30 35 400

0.05

0.1

0.15

0.2

0.25

0.3

0.35