Filtering and Identiﬁcation - Automatic Control · 2019-08-28 · 27/42 Division of Automatic Control - EE Observations from RangeDemo.m • columns of A lie in a plane ⊂ R3 ⇔

1/42

Division of Automatic Control - EE

Filtering and IdentificationDay 1 - Lecture 1:Introduction and refreshment LA

Michel Verhaegen

2/42


Smart Optics SystemsStar

Plane wavefront

Disturbed wavefront

Telescope /Collimator

Tip−tilt mirror

Beam splitter

Controller

Wavefrontsensor

Deformablemirror

Camera

Turbulent Atmosphere

Adaptive Optics Active correction of wavefront aberrations by a

deformable mirror. What is needed from a control engineer?

3/42


Lithography

Challenge: Aberration correction due to deformations in the mir-

rors caused by the heating of the Light Source (pm accuracy for

32nm technology! Internships possible!

4/42


Microscopy

Objective

Photon-

detector

Dichroic

beam splitter

Specimen

Confocal

pin hole

Lens

Laser

source

Focal plane

3D scanning

• Pin hole conjugated tothe focal point (rejectionof out-of-focus emission)

• 3-dimensional pointwisescanning (image formedby points)

• Confocal to widefield:

(Image courtesy: http://www.rudbeck.uu.se, AU: Airy-disk Unit)

5/42


Teaching Staff (DCSC)

Lecturer

Prof.dr.ir. Michel Verhaegen | [email protected]

Teaching Assistants (TA’s):

Jonas Calimer | [email protected]

Karl Granström | [email protected]

6/42


Objectives of the course

After studying this course you should

be able to derive estimation, filtering andidentification a algorithms based on the thelinear least squares method

aAnd control (H2, etc.)

7/42


Course material

• Book:Filtering and System Identification: AnIntroduction, by Michel Verhaegen andVincent Verdult, Cambridge University Press,2007.

• hand-outs or local blackboard?

8/42


Outline of the course

This intensive course will run for a week; withmorning lectures and homework in the afternoon.

• Day 1: LA review and Deterministic LinearLeast Squares

• Day 2: Stochastic Least Squares and Kalmanfiltering

9/42


Outline of the course (C’td)

• Day 3: Use of the Kalman filter and optimalpredictors for input-output models

• Day 4: Deterministic Subspace Identificationand a framework for consistency analysis

• Day 5: Instrumental variables in Subspaceidentification and probing some futuredevelopmentsNo Homework!

10/42


Exam

• Four sets of homeworks:Hand-in sets on morning of the next day tothe Lecturer.

11/42


Filtering and identificationLet’s start!

12/42


System identification?

in a general context

The art to extract missing information by inspectionwith the goal to ...

in a scientific context

The art to extract mathematical models from mea-surements derived by experimentation with physi-cal phenomenon one wants to understand/control(GOAL!)

13/42


Identification cycle

DATA GENERATION

MODEL VALIDATION

MODEL SELECTION AND ESTIMATION

USE THE MODEL

14/42


Mathematical ingredients?

(Linear) Least Squares

minx

ǫT ǫ y = Fx+ ǫ

• Matrix Theory

• Probability Theory

• Signal/System Theory

• Domain Knowledge

f1

f2

F

y

15/42


Overview Linear Algebra (LA)

• The matrix concept!• The Usefull matrix factorization: The SVD• A Quick view on its potential!• “Matrix-crimes”• The Useful matrix Lemma

16/42


Matrix theory: Some history

Matrix is Latin for womb (matrix = “mögel”,“grogrund”,matris)

Chinese used matrix methods already in [200 BC— 300AD].

1. They used concepts like determinants of atable of numbers

2. Determinant was long known to be inventedby Japanese Seki Kowa 1683.

17/42


Matrix theory: Some history

The term “Matrix” was first introduced byJames Sylvester 1850

history/

www−history.mcs.st−and.ac.uk/

Phil. Mag. S. 6, Vol 37, No. 251, Nov. 1850

Mr. J.J. Sylvesteron a new Class of Theorems

18/42


Definition of a matrixWhat it is not?

19/42


Definition of a matrix

A matrix A ∈ Rm×n is a

two-dimensional table of numbers:

A =

a11 a12 · · · a1n

a21 a22 a2n. . .

am1 am2 · · · amn

=

[

a1 a2 · · · an

]

with aij ∈ R, ai ∈ Rm.

20/42


A matrix represents a (linear) mapping

A matrix is (also) a mapping between twoEuclidean vector spaces:

A : Rn → Rm : ∀x ∈ R

n,∃y ∈ Rm : Ax = y

0 0

"A"RRn m

21/42


The “Four” key spaces of a linear mapping

0 0

"A"RRn m The linear mapping: A : R

n

(“domain”) → Rm (“Image or

Range space”) is character-ized by four subspaces:

• range(A) = {y ∈ Rm : y = Ax for some x ∈ R

n}

• range(AT ) = {x ∈ Rn : x = ATy for some y ∈ R

m}

• ker(A) = {x ∈ Rn : Ax = 0}

• ker(AT ) = {y ∈ Rm : ATy = 0}

The rank of A equals the dimension of range(A).

22/42


Special class of matricesDefinition: An “square” matrix Q ∈ R

n×n isorthogonal if

QTQ = QQT = In

This means:

1. Each column vector of an orthogonal matrix has length · · · ?

2. Two different column (row) vectors of an orthogonal matrix

satisfy?

3. What is the inverse of an orthogonal matrix?

4. And many more useful (numerical) advantages ...

23/42



• The matrix concept!• The Usefull matrix factorization: The SVD• A Quick view on its potential!• “Matrix-crimes”• The Useful matrix Lemma

24/42


The Singular value decomposition (SVD)The SVD-Theorem: Let A ∈ R

m×n, then there existsa pair of orthogonal matrices:

U =[

u1 · · · um

]

∈ Rm×m : UUT = UTU = Im

V =[

v1 · · · vn

]

∈ Rn×n : V V T = V TV = In

such that,

UTAV =

[

Σ 0

0 0

]

∈ Rm×n, Σ = diag(σ1, · · · , σp)

with σ1 ≥ σ2 ≥ · · · ≥ σp ≥ 0 and p = min(m,n).

25/42


Example SVDA =

1 0 1

1

2

1

21

0 1 1

⇒

A =

−√3

3

√2

2

√6

6

−√3

30

√6

3

−√3

3−

√2

2

√6

6

︸︷︷︸

U

3√2

20 0

0 1 0

0 0 0

︸︷︷︸

Σ

−√6

6

√2

2−

√3

3

−√6

6−

√2

2−

√3

3

−√6

30

√3

3

T

︸︷︷︸

V T

[U,Sigma,V]=svd(A);

• Column vectors of the matrix U : left singular vectors

• Column vectors of the matrix V : right singular vectors

• Diagonal elements of Σ: the singular values

26/42


RangeDemo.m

27/42


Observations from RangeDemo.m• columns of A lie in a plane ⊂ R

3 ⇔ dim(

spancol(A))

= 2 ⇔

# non-zero singular values (sv’s) = 2

• the left singular vectors u1, u2 corresponding to the non-zero

singular values:

A =

2∑

i=1

σiuivTi

form an orthogonal basis for spancol(A).

• the left singular vector u3 corresponding to the zero singular

value (i = 3) is a basis for ker(AT ).

• the left (and right) singular vectors are orthogonal and are of

unit length.

28/42


The four key subspaces

Let the SVD of the matrix A be given as,

A =[

U1 | U2

]

Σ1 | 0

0 | 0

V T1

V T2

with Σ1 > 0

then, since Ax =(U1

(Σ1(V

T1x)))

,

range(A) = {y ∈ Rm : y = Ax for some x ∈ R

n} = span(U1)

Further, since for x = V2α :

Ax = U1Σ1VT1V2α = 0,

ker(A) = {x ∈ Rn : Ax = 0} = span(V2)

0 0

"A"RRn m

V1

V2U1

U2

29/42


The SVD: the “workhorse” for reliable calculations

Contrary to the eigenvalue decomposition, thedeterminant, etc. the SVD allows for anumerically reliable “calculus”. Example:

Checking the singularity of a matrix A: Thenotion det(A) is “often” used to signal thesingularity of a matrix. This is only true in thecase it is “exactly” zero!

30/42


Checking Singularity (Ct’d)Example: Consider the “square” matrix:

A =

1 −1 · · · −1

0 1 · · · −1...

. . .

0 0 · · · 1

∈ Rn×n

Then det(A) equals 1. But the condition number of the matrix A

defined as:

κα(A) = ‖A‖α‖A−1‖α

for α = 1, 2,∞ and ‖A‖α = supx6=0

‖Ax‖α‖x‖α equals:

κ∞(A) = n2n−1

31/42


Condition number of a matrixDefinition: For a general matrix A ∈ R

m×n (m ≥ n), its condition

number κ2(A) (in short κ(A)) is given as:

κ(A) = ‖A‖2‖A†‖2

where A† denotes the pseudo-inverse of a matrix, i.e. satisfying,

AA†A = A A†AA† = A† (AA†)T = AA† (A†A)T = A†A

If A is full rank, then A† = (ATA)−1AT .

Exercise: Check that κ(A) = σ1

σn!

32/42



• The matrix concept!• The Usefull matrix factorization: The SVD• A Quick view on its potential• “Matrix-crimes”• The Useful matrix Lemma

33/42


“Optimal” low rank approximation

Theorem: Let the SVD in the SVD-theorem begiven and let k < rank(A) and let the followingapproximation Ak of A be given:

Ak =k∑

i=1

σiuivTi

then,

minrank(B)=k

‖A− B‖2 = ‖A− Ak‖2 = σk+1

34/42


Spiegelman.m

35/42



• The matrix concept!• The Usefull matrix factorization: The SVD• A Quick view on its potential• ‘‘Matrix-crimes”• The Useful matrix Lemma

36/42


Matrix-crimes: Syntax crimesa

1. Non-compatibility of dimensions: A+ B whenA ∈ R

2×3 and B ∈ R3×3 and the same for

ATB.

2. Matrix products do (in general) not commute:AB 6= BA.

3. Matrix inverse of the product of matrices:(AB)−1 6= A−1B−1 in stead of(AB)−1 = B−1A1 - provided inverses exist!

4. (A+ B)2 6= A2 + 2AB + B2!

aTypical violations of Stanford students [S. Boyd - EE 263], our TUD students

“too often” join the club ...

37/42


Matrix-crimes: Semantic crimes

Matrix expressions that simply do not makesense. Examples:

1. Let x ∈ Rn, then xxT exists but (xxT )−1 not,

why?

2. If the matrix Q ∈ Rm×n for m > n, then

QQT

can never be the identity matrix.

38/42



• The matrix concept!• The Usefull matrix factorization: The SVD• A Quick view on its potential• “Matrix-crimes”• The Useful matrix Lemma

39/42


Lemma 2.3 p. 19

Schur Complements: Block Triangular FactorizationsLet the block matrix A ∈ R

n×n (symmetric) be invertible, then a

very useful matrix factorization of matrix consisting of different

blocks is the following (C ∈ Rm×m):

A B

BT C

=

I 0

BTA−1 I

A 0

0 C −BTA−1B

I A−1B

0 I

Therefore the following holds,

A B

BT C

≥ 0 ⇔ A > 0 and C −BTA−1B ≥ 0

40/42


Exercise

Given

[

A B

BT C

]

(A ∈ Rn×nandC ∈ R

m×m)

(symmetric) with A > 0 and C − BTA−1B > 0,then show:

rank([

A B

BT C

])

= n+m

41/42


Lemma 2.3 p. 19

Schur Complements: Block Triangular FactorizationsWhen C is invertible, then we have:

A B

BT C

=

I BC−1

0 I

A−BC−1BT 0

0 C

I 0

C−1BT I

Therefore the following holds,

A B

BT C

≥ 0 ⇔ C > 0 and A−BC−1BT ≥ 0

The condition Matrix ≥ 0 among others means that a square

root of the matrix exists: Matrix = Matrix1/2MatrixT/2

42/42


Summary of Lecture 1

To start the discovery tour for retrieving system information from

measured data records:

What we just have done is a brief review of linear algebra. Next

we briefly review probability theory and filtering of stochastic

processes! We will also start with analysing the derterministic

least squares problem !

Reading of the course book of first Day Lecture:

Study Chapters 1, 2(2.1-2.5), 3, 4(4.1-4.3)

Documents

Filtering and Identiﬁcation - Automatic Control · 2019-08-28 · 27/42 Division of Automatic Control - EE Observations from RangeDemo.m • columns of A lie in a plane ⊂ R3 ⇔