Chapter 1 Numerical approximation of data: interpolation, least squares methodmath.univ-lille1.fr/~debievre/M45website/chapter_interp... · 2011-01-18 · Chapter 1 Numerical approximation

Chapter 1

Numerical approximation of data :

interpolation, least squares method

I. Motivation

1 Approximation of functions

Evaluation of a function

Which functions (f : R→ R) can be effectively evaluated inany point ?

Þ the power functions : f(x) = xm, m ∈ NÞ the polynomial functions :

f(x) = a0 + a1x+ a2x2 + · · ·+ amx

m

How can we evaluate other functions in a given point ?

for instance : f(x) = cos(x), f(x) = sin(x) exp(x),...

Þ approximation by a polynomial function :

H using a Taylor series about the given point,

H searching a polynomial having the same values as the

function in some close points

á Lagrange interpolation




f(x) = a0 + a1x+ a2x2 + · · ·+ amx

m











f(x) = a0 + a1x+ a2x2 + · · ·+ amx

m











f(x) = a0 + a1x+ a2x2 + · · ·+ amx

m








Principles of Lagrange interpolation

f(x) = sin(πx

2)(x2 + 3)


f(x) = sin(πx

2)(x2 + 3)

4 points on the curve :

(−1,−4),(1, 4),(2, 0),

(3,−12)


f(x) = sin(πx

2)(x2 + 3) Lagrange interpolating polynomial

4 points on the curve : P polynomial of degree ≤ 3

satisfying

(−1,−4), P (−1) = −4(1, 4), P (1) = 4(2, 0), P (2) = 0

(3,−12) P (3) = −12

Principles of Lagrange interpolation, with 6 points

f(x) = sin(πx

2)(x2 + 3) Lagrange interpolating polynomial

6 points on the curve : P polynomial of degree ≤ 5

satisfying

(xi, yi)0≤i≤5, P (xi) = yi for 0 ≤ i ≤ 5

Remark : ouside the interval defined by the (xi), the Lagrangeinterpolating polynomial has nothing to do with f .

I. Motivation

1 Approximation of functions2 Curve approximation

Piecewise interpolation

f(x) = sin(πx

2)(x2 + 3)

piecewise affine approximation


f(x) = sin(πx

2)(x2 + 3) piecewise affine approximation


f(x) = sin(πx


Applications : Calculation of an approximate value of

the length of the curve

the area under the curve (here

∫ 5

−5f(x)dx)

Þ see chapter 2


f(x) = sin(πx


Applications : Calculation of an approximate value of

the length of the curve

the area under the curve (here

∫ 5

−5f(x)dx)

Þ see chapter 2

Cubic spline

f(x) = sin(πx

2)(x2 + 3) s cubic spline

Principle :

between two consecutive points, s is a cubic polynomial

s(xi) = f(xi)

s ∈ C2

+ two conditions on the boundary points

Cubic spline

f(x) = sin(πx

2)(x2 + 3) s cubic spline

Principle :

between two consecutive points, s is a cubic polynomial

s(xi) = f(xi)

s ∈ C2

+ two conditions on the boundary points

Bezier curves : principle

P0, P1, · · · , Pn, are n+ 1 given control points

The corresponding Bezier curve is defined by

M(t) =

n∑i=0

Bin(t)Pi, 0 ≤ t ≤ 1

where Bin are Bernstein polynomial defined by

Bin = CinX

i(1−X)n−i, with Cin =n!

i!(n− i)!.

Bezier curves : principle

P0, P1, · · · , Pn, are n+ 1 given control points

The corresponding Bezier curve is defined by

M(t) =

n∑i=0

Bin(t)Pi, 0 ≤ t ≤ 1

where Bin are Bernstein polynomial defined by

Bin = CinX

i(1−X)n−i, with Cin =n!

i!(n− i)!.

Bezier curves : with more points

Application of the Bezier curves

I. Motivation

1 Approximation of functions2 Curve approximation3 Fitting of statistical data

Study of statistical data

Some experimental measurements

H X : noise level in the factory (in dB),

H Y : time used to do a definite work (in minutes)

X 73 78 76 63 81 70 75 81 79 84 50 76 65 58Y 77 85 79 67 83 73 72 83 81 82 52 77 65 58

A physical measure alwayscontains some noise. Canwe find a law linking Y andX (Y = f(X)) ?

Can we predict the value ofY for X = 66dB ?

Linear regression

Principle

We are looking for a and b such that

d(a, b) =∑

(yi − (axi + b))2 is minimal.

The straight line y = ax+ b is the linear regression line.

The polynomial P = aX + b is the least squares fittingpolynomial of the cloud of points.

II. Lagrange interpolating polynomial :

theoretical study

1 Study of an example

Lagrange interpolation in 1 point

f(x) = sin(πx

2)(x2 + 3)

1 point on the curve : Search for P0, such that(x0, y0) = (1, 4) degP0 ≤ 0 and

P0(x0) = y0

P0 = 4


f(x) = sin(πx

2)(x2 + 3)


P0(x0) = y0

P0 = 4


f(x) = sin(πx

2)(x2 + 3)


P0(x0) = y0

P0 = 4

Lagrange interpolation in 2 points

f(x) = sin(πx

2)(x2 + 3)

2 points on the curve : Search for P1, such that(x0, y0) = (1, 4) degP1 ≤ 1 and

(x1, y1) = (−1,−4) P1(x0) = y0 and P1(x1) = y1

P1 = 4X


f(x) = sin(πx

2)(x2 + 3)


(x1, y1) = (−1,−4) P1(x0) = y0 and P1(x1) = y1

P1 = 4X


f(x) = sin(πx

2)(x2 + 3)


(x1, y1) = (−1,−4) P1(x0) = y0 and P1(x1) = y1

P1 = 4X


f(x) = sin(πx

2)(x2 + 3)


(x1, y1) = (−1,−4) P2(x0) = y0, P2(x1) = y1(x2, y2) = (3,−12) and P2(x2) = y2


f(x) = sin(πx

2)(x2 + 3)


(x1, y1) = (−1,−4) P2(x0) = y0, P2(x1) = y1(x2, y2) = (3,−12) and P2(x2) = y2


Idea

Search for L0, such that degL0 = 2 and

L0(x1) = L0(x2) = 0 and L0(x0) = 1.


L1(x0) = L1(x2) = 0 and L1(x1) = 1.


L2(x0) = L2(x1) = 0 and L2(x2) = 1.

Prove that P2 = y0L0 + y1L1 + y2L2 is a solution of the pb.

Give the expression of P .

á P2 = −3X2 + 4X + 3


Idea


L0(x1) = L0(x2) = 0 and L0(x0) = 1.


L1(x0) = L1(x2) = 0 and L1(x1) = 1.


L2(x0) = L2(x1) = 0 and L2(x2) = 1.

Prove that P2 = y0L0 + y1L1 + y2L2 is a solution of the pb.

Give the expression of P .

á P2 = −3X2 + 4X + 3


Other idea

P1 = 4X satisfies P1(1) = 4, P1(−1) = −4 and degP1 = 1.

Q = (X + 1)(X − 1) satisfies Q(1) = Q(−1) = 0 anddegQ = 2 (in fact Q = −L2)

Search for P2 under the form

P2 = P1 + αQ = 4X + α(X + 1)(X − 1).

P2(3) = −12⇐⇒ α = −3 and

P2 = 4X − 3(X + 1)(X − 1) = −3X2 + 4X + 3


Other idea

P1 = 4X satisfies P1(1) = 4, P1(−1) = −4 and degP1 = 1.

Q = (X + 1)(X − 1) satisfies Q(1) = Q(−1) = 0 anddegQ = 2 (in fact Q = −L2)

Search for P2 under the form

P2 = P1 + αQ = 4X + α(X + 1)(X − 1).

P2(3) = −12⇐⇒ α = −3 and

P2 = 4X − 3(X + 1)(X − 1) = −3X2 + 4X + 3


theoretical study

1 Study of an example2 Existence and uniqueness of the Lagrange interpolating

polynomial

The mathematical problem

Formulation

Let (n+ 1) points be given :

(xi, yi)0≤i≤n with (xi, yi) ∈ R2 for all i,

xi 6= xj for all i 6= j.

Is it possible to find a polynomial P with real coefficients satisfying

P (xi) = yi ∀0 ≤ i ≤ n?

Degree of P ?

number of equations : n+ 1

Þ number of unknowns (coefficients (ai)) less that n+ 1

Þ degP ≤ n

The Lagrange basis

For 0 ≤ j ≤ n, let us define

Lj =

n∏i=0,i 6=j

X − xixj − xi

.

It satisfies :

Lj(xj) = 1 and Lj(xi) = 0 for all i 6= j

⇐⇒ Lj(xi) = δi,j .

anddegLj = n ∀0 ≤ j ≤ n.

Solution of the problem + uniqueness

Existence of a solution

Let P =

n∑j=0

yjLj , we have

degP ≤ n,

P (xi) =

n∑j=0

yjLj(xi) = yi for all 0 ≤ i ≤ n.

á P is a solution of the Lagrange interpolation problem.

Uniqueness

Can we find another solution to the problem, Q ? If Q exists,

degQ ≤ n and deg(P −Q) ≤ n,

P (xi)−Q(xi) = (P −Q)(xi) = 0 for 0 ≤ i ≤ n.

á P −Q = 0 and the solution is unique.

Main theorem

Theorem

Hypotheses :

Let us consider (n+ 1) points of R2 : (xi, yi)0≤i≤n,

xi 6= xj for all i 6= j

Then, there exists a unique polynomial P ∈ Rn[X] satisfying

P (xi) = yi ∀0 ≤ i ≤ n.

P is the Lagrange interpolating polynomial that passes through the(n+ 1) points (xi, yi)0≤i≤n.

In the case where yi = f(xi) for all 0 ≤ i ≤ n (with f a givenfunction), P is the Lagrange interpolating polynomial of f in thepoints (xi)0≤i≤n.


theoretical study

1 Study of an example2 Existence and uniqueness of the Lagrange interpolating

polynomial3 Interpolation error result

Presentation of the problem

Comparison of f and P (4 points)

E(x) = f(x)− P (x)



E(x) = f(x)− P (x)

with a zoom around the points



E(x) = f(x)− P (x)



E(x) = f(x)− P (x)

with a zoom around the points

What can be said about E(x) ? Can it be bounded ?...

Interpolation error

Theorem

Hypotheses :

f : [a, b]→ R, f ∈ Cn+1([a, b]),

(xi)0≤i≤n, n+ 1 distinct real numbers of [a, b].

Pn : Lagrange interpolating polynomial of f in the points (xi)0≤i≤n.

Then, for all x ∈ [a, b], there exists ξx ∈ [a, b] such that

f(x)− Pn(x) =1

(n+ 1)!Πn(x)f (n+1)(ξx),

with Πn =n∏i=0

(X − xi).

Consequence

As a consequence, we get :

∀x ∈ [a, b] |f(x)− Pn(x)| ≤ 1

(n+ 1)!Mn+1|Πn(x)|,

with Mn+1 = maxξ∈[a,b]

|f (n+1)(ξ)|.

It does not imply the convergence of Pn(x) towards f(x).

It is not necessary interesting to increase n.

III. Lagrange interpolating polynomial :

practical computation

1 Cost of the computation of the interpolating polynomial

With the Lagrange basis

Lj(x) =

n∏i=0,i 6=j

(x− xi)

n∏i=0,i 6=j

(xj − xi),

for j = 0 to n

Cost of the computation :

(n+ 1)×

(2(n− 1) mult. + 1 div.

)

P (x) =n∑j=0

yjLj(x) Þ final cost ≈ 2n2.

Other main drawback of the method : what happens if wefinally want to add one more point (xn+1, yn+1) ?

Þ All must be started again from zero.


Lj(x) =

n∏i=0,i 6=j

(x− xi)

n∏i=0,i 6=j

(xj − xi), for j = 0 to n

Cost of the computation : (n+ 1)×(

2(n− 1) mult. + 1 div.)

P (x) =n∑j=0





Lj(x) =

n∏i=0,i 6=j

(x− xi)

n∏i=0,i 6=j



2(n− 1) mult. + 1 div.)

P (x) =n∑j=0





Lj(x) =

n∏i=0,i 6=j

(x− xi)

n∏i=0,i 6=j



2(n− 1) mult. + 1 div.)

P (x) =n∑j=0




With an other basis

Idea

Write the polynomial in the basis1, (X − x0), (X − x0)(X − x1), · · · ,n−1∏j=0

(X − xj)

.

Þ P

n

= α0 + α1(X − x0) + · · ·+ αn

n−1∏j=0

(X − xj).

Now, if we add one point (xn+1, yn+1) , we have :

Pn+1 = Pn + αn+1

n∏j=0

(X − xj)

Þ we just need to calculate αn+1 to get Pn+1.

With an other basis

Idea


(X − xj)

.

Þ Pn = α0 + α1(X − x0) + · · ·+ αn

n−1∏j=0

(X − xj).


Pn+1 = Pn + αn+1

n∏j=0

(X − xj)


With an other basis

Idea


(X − xj)

.

Þ Pn = α0 + α1(X − x0) + · · ·+ αn

n−1∏j=0

(X − xj).


Pn+1 = Pn + αn+1

n∏j=0

(X − xj)


Cost of the computation of Pn(x)

Pn(x) = α0 + α1(x− x0) + · · ·+ αn

n−1∏j=0

(x− xj)

= α0 + (x− x0)(α1 + (x− x1)

(α2 +

(· · ·)))

.

Horner’s algorithm for the computation of p = Pn(x)

p← αn

for k from n− 1 to 0

p← αk + (x− xk)pend

Cost

n additions + n multiplications

Þ and the computation of the coefficients αi ?

Cost of the computation of Pn(x)

Pn(x) = α0 + α1(x− x0) + · · ·+ αn

n−1∏j=0

(x− xj)

= α0 + (x− x0)(α1 + (x− x1)

(α2 +

(· · ·)))

.

Horner’s algorithm for the computation of p = Pn(x)

p← αn

for k from n− 1 to 0

p← αk + (x− xk)pend

Cost

n additions + n multiplications

Þ and the computation of the coefficients αi ?

III. Lagrange interpolating polynomial :

practical computation

1 Cost of the computation of the interpolating polynomial

2 The divided difference method

Calculation of the first αi

n = 0, 1 first point (x0, y0), P0 = α0 :

P0(x0) = y0 =⇒ α0 = y0

n = 1, + (x1, y1), P1 = y0 + α1(X − x0) :

P1(x1) = y1 =⇒ α1 =y1 − y0x1 − x0

n = 2, + (x2, y2),

P2 = y0 +y1 − y0x1 − x0

(X − x0) + α2(X − x0)(X − x1)

P2(x2) = y2 =⇒ α2 =

y2 − y1x2 − x1

− y1 − y0x1 − x0

x2 − x0

Recurrence formula

Assume we have :

Interpolating points : (x0, y0) · · · (xn−1, yn−1),

Pn−1 = α0 +

n−1∑j=1

αj

j−1∏k=0

(X − xk), (αj)0≤j≤n−1 known

Interpolating points : (x1, y1) · · · (xn, yn),

Qn−1 = β0 +

n−1∑j=1

βj

j∏k=1

(X − xk), (βj)0≤j≤n−1 known

Then,X − x0xn − x0

Qn−1 +xn −Xxn − x0

Pn−1

= Pn

and

αn =1

xn − x0βn−1 −

1

xn − x0αn−1 =

βn−1 − αn−1xn − x0

.

Recurrence formula

Assume we have :


Pn−1 = α0 +

n−1∑j=1

αj

j−1∏k=0



Qn−1 = β0 +

n−1∑j=1

βj

j∏k=1




Pn−1 = Pn

and

αn =1

xn − x0βn−1 −

1

xn − x0αn−1 =

βn−1 − αn−1xn − x0

.

Recurrence formula

Assume we have :


Pn−1 = α0 +

n−1∑j=1

αj

j−1∏k=0



Qn−1 = β0 +

n−1∑j=1

βj

j∏k=1




Pn−1 = Pn

and

αn =1

xn − x0βn−1 −

1

xn − x0αn−1 =

βn−1 − αn−1xn − x0

.

The divided differences

x0 f(x0)

x1 f(x1)

x2 f(x2)

......

......

. . .

xn−1 f(xn−1)

. . .

xn f(xn)

· · · · · · f [x0, · · · , xn]

with

f [x0, · · · , xn] =f [x1, · · · , xn]− f [x0, · · · , xn−1]

xn − x0.


x0 f [x0]

x1 f [x1]

x2 f [x2]

......

......

. . .

xn−1 f [xn−1]

. . .

xn f [xn]

· · · · · · f [x0, · · · , xn]

with

f [x0, · · · , xn] =f [x1, · · · , xn]− f [x0, · · · , xn−1]

xn − x0.


x0 f [x0]

x1 f [x1]f [x1]− f [x0]

x1 − x0

x2 f [x2]f [x2]− f [x1]

x2 − x1...

......

.... . .

xn−1 f [xn−1]f [xn−1]− f [xn−2]

xn−1 − xn−2

. . .

xn f [xn]f [xn]− f [xn−1]

xn − xn−1

· · · · · · f [x0, · · · , xn]

with

f [x0, · · · , xn] =f [x1, · · · , xn]− f [x0, · · · , xn−1]

xn − x0.


x0 f [x0]

x1 f [x1] f [x0, x1]

x2 f [x2] f [x1, x2]

......

...

.... . .

xn−1 f [xn−1] f [xn−2, xn−1]

. . .

xn f [xn] f [xn−1, xn]

· · · · · · f [x0, · · · , xn]

with

f [x0, · · · , xn] =f [x1, · · · , xn]− f [x0, · · · , xn−1]

xn − x0.


x0 f [x0]

x1 f [x1] f [x0, x1]

x2 f [x2] f [x1, x2]f [x1, x2]− f [x0, x1]

x2 − x0...

......

...

. . .

xn−1 f [xn−1] f [xn−2, xn−1]f [xn−1, xn]− f [xn−2, xn−1]

xn − xn−2

. . .

xn f [xn] f [xn−1, xn]f [xn−1, xn]− f [xn−2, xn−1]

xn − xn−2

· · · · · · f [x0, · · · , xn]

with

f [x0, · · · , xn] =f [x1, · · · , xn]− f [x0, · · · , xn−1]

xn − x0.


x0 f [x0]

x1 f [x1] f [x0, x1]

x2 f [x2] f [x1, x2] f [x0, x1, x2]

......

......

. . .

xn−1 f [xn−1] f [xn−2, xn−1] f [xn−3, xn−2, xn−1]

. . .

xn f [xn] f [xn−1, xn] f [xn−2, xn−1, xn]

· · · · · · f [x0, · · · , xn]

with

f [x0, · · · , xn] =f [x1, · · · , xn]− f [x0, · · · , xn−1]

xn − x0.


x0 f [x0]

x1 f [x1] f [x0, x1]

x2 f [x2] f [x1, x2] f [x0, x1, x2]

......

......

. . .

xn−1 f [xn−1] f [xn−2, xn−1] f [xn−3, xn−2, xn−1]. . .

xn f [xn] f [xn−1, xn] f [xn−2, xn−1, xn] · · · · · · f [x0, · · · , xn]

with

f [x0, · · · , xn] =f [x1, · · · , xn]− f [x0, · · · , xn−1]

xn − x0.


x0 f [x0]

x1 f [x1] f [x0, x1]

x2 f [x2] f [x1, x2] f [x0, x1, x2]

......

......

. . .



Þ Pn = f [x0]+f [x0, x1](X−x0)+f [x0, x1, x2](X−x0)(X−x1)

+ · · ·+ f [x0, · · · , xn]

n−1∏j=0

(X − xj).


x0 f [x0]

x1 f [x1] f [x0, x1]

x2 f [x2] f [x1, x2] f [x0, x1, x2]

......

......

. . .



Cost of the computation ≈ n2

2div. and n2 sub.

Result

Theorem

The Lagrange interpolating polynomial of f in the points (xi)0≤i≤nreads

Pn = f [x0] +

n∑j=1

f [x0, · · · , xj ]j−1∏k=0

(X − xk),

where f [ ] denotes the divided difference of f defined by induction

f [xi] = f(xi) for 0 ≤ i ≤ n

f [xi, · · · , xi+k] =f [xi+1, · · · , xi+k]− f [xi, · · · , xi+k−1]

xi+k − xifor 0 ≤ i ≤ n− k, 1 ≤ k ≤ n.

IV. A few words about Hermite interpolation

1 Presentation of the problem

One example

f(x) = sin(πx

2)(x2 + 3)

Consider two points :

(x0, y0) = (−1,−4)

(x1, y1) = (3,−12)

P = −4− 2(X + 1) = −6− 2X

Q = P + (X + 1)(X − 3)(αX + β)

= P + (X + 1)(X − 3)(−1)

= −X2 − 3

One example

f(x) = sin(πx

2)(x2 + 3)


(x0, y0) = (−1,−4)

(x1, y1) = (3,−12)

Search P such that

P (x0) = f(x0), P (x1) = f(x1)

P = −4− 2(X + 1) = −6− 2X

Q = P + (X + 1)(X − 3)(αX + β)

= P + (X + 1)(X − 3)(−1)

= −X2 − 3

One example

f(x) = sin(πx

2)(x2 + 3)


(x0, y0) = (−1,−4)

(x1, y1) = (3,−12)

Search P such that

P (x0) = f(x0), P (x1) = f(x1)

P = −4− 2(X + 1) = −6− 2X

Q = P + (X + 1)(X − 3)(αX + β)

= P + (X + 1)(X − 3)(−1)

= −X2 − 3

One example

f(x) = sin(πx

2)(x2 + 3)


(x0, y0) = (−1,−4)

(x1, y1) = (3,−12)

Search Q such that

Q(x0) = f(x0), Q(x1) = f(x1)

Q′(x0) = f ′(x0), Q′(x1) = f ′(x1)

P = −4− 2(X + 1) = −6− 2X

Q = P + (X + 1)(X − 3)(αX + β)

= P + (X + 1)(X − 3)(−1)

= −X2 − 3

One example

f(x) = sin(πx

2)(x2 + 3)


(x0, y0) = (−1,−4)

(x1, y1) = (3,−12)

Search Q such that

Q(x0) = f(x0), Q(x1) = f(x1)

Q′(x0) = f ′(x0), Q′(x1) = f ′(x1)

P = −4− 2(X + 1) = −6− 2X

Q = P + (X + 1)(X − 3)(αX + β)

= P + (X + 1)(X − 3)(−1)

= −X2 − 3

One example

f(x) = sin(πx

2)(x2 + 3)


(x0, y0) = (−1,−4)

(x1, y1) = (3,−12)

Search Q such that

Q(x0) = f(x0), Q(x1) = f(x1)

Q′(x0) = f ′(x0), Q′(x1) = f ′(x1)

P = −4− 2(X + 1) = −6− 2X

Q = P + (X + 1)(X − 3)(αX + β)

= P + (X + 1)(X − 3)(−1)

= −X2 − 3

With 2 other points

f(x) = sin(πx

2)(x2 + 3)

P = 0

Q = −π2X3 +

π

4X2 +

3π

2X

The mathematical problem

Generalities

The Hermite interpolation takes into account

the values of the function in some points (xi)0≤i≤k,

the values of the successive derivatives of the function untilorder αi in xi.

Formulation

f is a sufficiently smooth function defined on [a, b],

x0, . . . , xk are (k + 1) given points of [a, b],

α0, . . . , αk are (k + 1) integers.

Is it possible to find P satisfying

∀0 ≤ i ≤ k, P (j)(xi) = f (j)(xi),∀0 ≤ j ≤ αi?

IV. A few words about Hermite interpolation

1 Presentation of the problem

2 Main results

Analysis of the problem

Degree of P

Number of equations :

k∑i=0

(αi + 1) = k + 1 +

k∑i=0

αi.

Degree : n = k +

k∑i=0

αi.

Definition

P is the Hermite interpolating polynomial of f in thepoints (xi)0≤i≤k with the orders (αi)0≤i≤k

Theorem

Theorem : existence and uniqueness + interpolation error

Hypotheses :

(xi)0≤i≤k, (k + 1) points in [a, b],

(αi)0≤i≤k, (k + 1) integers,n = k +

k∑i=0

αi

f : [a, b]→ R, f ∈ Cn+1([a, b]),

Then, there exists a unique polynomial Pn ∈ Rn[X] such that

∀0 ≤ i ≤ k, P (j)n (xi) = f (j)(xi), ∀0 ≤ j ≤ αi.

Furthermore, for all x ∈ [a, b], there exists ξx ∈ [a, b] such that

f(x)− Pn(x) =1

(n+ 1)!Ωn(x)f (n+1)(ξx),

with Ωn =k∏i=0

(X − xi)αi+1.

V. Least squares method

1 The case of linear regression

Linear regression

We are looking for a0 and b0 suchthat the following distance is mini-mal :

d(a, b) =

n∑i=1

(yi − (axi + b))2.

Necessary condition

∂d

∂a(a0, b0) = 0 and

∂d

∂b(a0, b0) = 0

⇐⇒

a0

n∑i=1

x2i + b0

n∑i=1

xi =

n∑i=1

xiyi

a0

n∑i=1

xi + b0 n =

n∑i=1

yi

Existence of a unique candidate (a0, b0)

Matrix of the linear system

A =

n∑i=1

x2i

n∑i=1

xi

n∑i=1

xi n

= n

1

n

n∑i=1

x2i X

X 1

Invertibility

detA = n2(1

n

n∑i=1

x2i − X2) = n

n∑i=1

(xi − X)2 = n2V(X).

ConclusionAs soon as two xi are different, detA 6= 0 and there exists aunique (a0, b0) susceptible to be a minimum of d :

a0 =Cov(X,Y )

V(X)and b0 = Y − a0X.

(a0, b0) is a minimizer of d

After some computations, we prove that

d(a, b)− d(a0, b0) =

n∑i=1

((a0 − a)xi + b0 − b)2

It yields∀(a, b) ∈ R2 d(a, b) ≥ d(a0, b0).

Remark on the matrix A

A =

n∑i=1

x2i

n∑i=1

xi

n∑i=1

xi n

= BTB with B =

x1 1x2 1...

...xn 1

V. Least squares method

1 The case of linear regression

2 Generalization


Given points

The cloud of points is still given by (xi)1≤i≤n and (yi)1≤i≤n.

A space of functionsFor some independent functions (ϕ1, · · · , ϕm), let us define

U = ϕ; ϕ =

m∑i=1

uiϕi

Search for a minimizer

We are looking for ϕ∗ ∈ U such that

n∑i=0

|yi − ϕ∗(xi)|2 = minϕ∈U

n∑i=0

|yi − ϕ(xi)|2.

Main result

Theorem

As soon as two xi are different, the least squares problem admits aunique solution

ϕ∗ =m∑i=1

u∗iϕi.

Futhermore, the vector u∗ = (u∗1, · · · , u∗m) is the unique solution ofthe linear system

BTBu∗ = BT y,

with

B =

ϕ1(x1) · · · ϕm(x1)...

...ϕ1(xn) · · · ϕm(xn)

RemarkIn the linear case, m = 2 and ϕ1(x) = x, ϕ2(x) = 1.

Documents

Chapter 1 Numerical approximation of data: interpolation, least squares methodmath.univ-lille1.fr/~debievre/M45website/chapter_interp... · 2011-01-18 · Chapter 1 Numerical approximation