ML_ICA

7/23/2019 ML_ICA

http://slidepdf.com/reader/full/mlica 1/23

Independent Component Analysis &

Blind Source Separation

Ata Kaban

The University of Birmingham

7/23/2019 ML_ICA


Overvie

! Today e learn about

" The coc#tail party problem $$ called also %blind

source separation 'BSS( " Independent Component Analysis 'ICA( for solving

BSS

" Other applications of ICA ) BSS

! At an intuitive & introductory & practical level

7/23/2019 ML_ICA


A bit li#e*

in the sense of having to find +uantities that

are not observable directly

7/23/2019 ML_ICA


Signals, -oint density

time

A m p l i t u

d e

S . ' t (

A m

p l i t u d e

S / ' t (

Signals 0oint density

s

marginal

densities

7/23/2019 ML_ICA


Original signals (hidden sources)s1(t), s2(t), s3(t), s4(t), t=1:T

7/23/2019 ML_ICA


The ICA model

s. s/

s1 s2

3. 3/ 31 32

a..

a./a.1

a.2

3i't( 4 ai.5s.'t( 6

ai/5s/'t( 6

ai15s1't( 6ai25s2't(

7ere, i4.829

In vector$matri3notation, and dropping

inde3 t, this is

x 4 A 5 s

7/23/2019 ML_ICA


This is recorded by the microphones: alinear mixture of the sources

xi(t) = ai1*s1(t) + ai2*s2(t) + ai3*s3(t) + ai4*s4(t)

7/23/2019 ML_ICA


The coctail party problem

Called also Blind Source Separation 'BSS( problemIll posed problem, unless assumptions are made:

The most common assumption is that source signals are

statistically independent9 This means that #noing the value

of one of them does not give any information about the other9

The methods based on this assumption are called IndependentComponent Analysis methods9 These are statistical

techni+ues of decomposing a comple3 data set into

independent parts9

It can be shown that under some reasonable conditions, if the

ICA assumption holds, then the source signals can berecovered up to permutation and scaling.

Determine the source signals, givenonly the mixtures

7/23/2019 ML_ICA


Recovered signals

7/23/2019 ML_ICA


Some further considerations

! If e #ne the mi3ing parameters ai- then e ould

-ust need to solve a linear system of e+uations9

!;e #no neither ai- nor si9

! ICA as initially developed to deal ith problems

closely related to the coctail party problem

! <ater it became evident that ICA has many other

applications too9 =9g9 from electrical recordings ofbrain activity from different locations of the scalp

'==> signals( recover underlying components of

brain activity

7/23/2019 ML_ICA


Illustration of ICA ith / signals

s.

s/

3.

3/

T t

t sat sat x

t sat sat x

:1

)()()(

)()()(

2221212

2121111

=∀

+=

+=

Original s ?i3ed signals

a2

a1

a1

7/23/2019 ML_ICA



3.

3/

T t

t sat sat x

t sat sat x

:1

)()()(

)()()(

2221212

2121111

=∀

+=

+=

Step.8Sphering

Step/8

@otatation

?i3ed signals

a2

a1

a1

7/23/2019 ML_ICA



s.

s/

3.

3/

T t

t sat sat x

t sat sat x

:1

)()()(

)()()(

2221212

2121111

=∀

+=

+=

Step.8Sphering

Step/8

@otatation

Original s ?i3ed signals

a2

a1

a1

7/23/2019 ML_ICA


=3cluded case

There is one case hen rotation

doesnt matter9 This case cannot be

solved by basic ICA9

*hen both densities are

>aussian

=3ample of non$>aussian density '$(

vs9>aussian '$9(

See# non$>aussian sources for to reasons8

5 identifiability

5 interestingness8 >aussians are not

interesting since the superposition of

independent sources tends to be >aussian

7/23/2019 ML_ICA


Computing the pre$processing steps for ICA

( Centring 4 ma#e the signals centred in ero

3i 3i $ =3iD for each i

.( Sphering 4 ma#e the signals uncorrelated9 I9e9 apply a transform V tox such that Cov'Vx(4I )) here Cov'y(4=yyTD denotes covariancematri3

V4=xxTD$.)/ )) can be done using %s+rtm function in ?at<ab

xVx )) for all t 'inde3es t dropped here(

)) bold loercase refers to column vectorE bold upper to matri3

Scope8 to ma#e the remaining computations simpler9 It is #non thatindependent variables must be uncorrelated " so this can be fulfilledbefore proceeding to the full ICA

7/23/2019 ML_ICA


Fi3ed Goint Algorithm

Input8 X

@andom init of W

Iterate until convergence8

Output8 W, S

1)(

)(

−

=

=

=

WWWW

SXW

XWS

T

T

T

g

∑=

−−=T

t

T

t

T GObj

1

)()()( IWWΛxWW

0ΛWXWXW

=−=∂

∂ T T g Obj

)(

here g'9( is derivative of >'9(,

W is the rotation transform sought Λ is <agrange multiplier to enforce that

; is an orthogonal transform i9e9 a rotation

Solve by fi3ed point iterations

The effect of Λ is an orthogonal de$correlation

Aapo 7yvarinen 'H(

Computing the rotation step

This is based on an the ma3imisation of an

ob-ective function >'9( hich contains an

appro3imate non$>aussianity measure9

The overall transform

then to ta#e X bac# to S is

'WTV(

There are several g'9(

options, each ill or# best

in special cases9 See

FastICA s ) tut for details9

7/23/2019 ML_ICA


Application domains of ICA

! Blind source separation (Bell&Sejnowski !e won "ee#irola$i %yarinen etc'

! I$a)e denoisin) (%yarinen

! *edical si)nal processin) + ,*-I ./# ..# (*ackei)

! *odellin) o, t0e 0ippoca$pus and isual cortex ("orinc%yarinen

! eature extraction ,ace reco)nition (*arni Bartlett

! /o$pression redundancy reduction

! Water$arkin) (3 "owe

! /lusterin) (#irola$i 4olenda

! !i$e series analysis (Back Valpola

! !opic extraction (4olenda Bin)0a$ 4a5an

! Scienti,ic 3ata *inin) (4a5an etc

7/23/2019 ML_ICA


Image denoising

;ienerfiltering

ICAfiltering

Joisy

imageOriginal

image

7/23/2019 ML_ICA


Clustering

In multi$variate data

search for the direction

along of hich the

pro-ection of the data is

ma3imally non$>aussian 4

has the most %structure

7/23/2019 ML_ICA


Blind Separation of Information from >ala3y

Spectra

0 50 100 150 200 250 300 350-0.2

0

0.2

0.4

0.6

0.8

1

1.2

1.4

7/23/2019 ML_ICA


ecomposition using Ghysical ?odels

ecomposition using ICA

7/23/2019 ML_ICA


Summing Up

! Assumption that the data consists of un#noncomponents " Individual signals in a mi3

" topics in a te3t corpus

" basis$gala3ies

! Trying to solve the inverse problem8

" Observing the superposition only " @ecover components

" Components often give simpler, clearer vie of thedata

7/23/2019 ML_ICA


@elated resources

http8))9cis9hut9fi)pro-ects)ica)coc#tail)coc#tailLen9cgi

emo and lin#s to further info on ICA9

http8))9cis9hut9fi)pro-ects)ica)fastica)code)dlcode9shtmlICA softare in ?at<ab9

http8))9cs9helsin#i9fi)u)ahyvarin)papers)JJne9pdf

Comprehensive tutorial paper, slightly more technical9

Documents

ML_ICA