An Introduction of Independent Component

Analysis (ICA)

Xiaoling WangJan. 28, 2003

2

What Is ICA?Application: blind source separation (BSS) and deconvolutionMotivation: “cocktail party problem” Assumption: two people speaking

simultaneously, two microphones in different locations

2221212

2121111

)()(

sasatxsasatx

3

Principles of ICA AlgorithmAssumption: sources are statistically independentGoal: it seeks a transformation to coordinates in which the data are maximally statistically independentDefinition: )()(

)()(tWxtytAstx

Mixing process

Demixing process – mixing matrix, – separation matrixA W

4

Hierarchy of ICA ModelsNonlinear mixing

nsfx )(

Linear mixingnAsx

Classical ICA

FlexibleSource model

Infomax

Non-stationarysources

Non-stationarymixing)(tAA

No noise

Asx Factor Analysis

R diagonal

Gaussian sources

IndependentFactor analysis

Non-Gaussian sources

Cumulant basedmethods

Approximations to

mutual information Switching

source modelProbabilistic

PCA

FastICA

Kurtosisminimizatio

n

Fixedsource model PCA

orthogonal mixing

No noise

5

Independence of SourcesIndependence: the pdf of sources can be factorized

Nongaussian is independentSeek the separation matrix W which maximize the nongaussianity of the estimated sources

M

mm tspSp

1

))(()(

6

Measures of Nongaussianity

Kurtosis (4th order cumulant):

Subgaussian: negative kurtosis Supergaussian: positive kurtosisNegentropy:

224 }){(3}{)( yEyEykurt

)()()(

)(log)()(

yHyHyJ

dyyfyfyH

gauss

differentialentropy

entropy i

ii aYpaYpYH )(log)()(

negentropy

7

Measures of Nongaussianity (Cont.)

Mutual information:

m

iim YHyHyyI

11 )()(),...,(

For , i

im WXHyHyyI detlog)()(),...,( 1WXY

8

FastICA AlgorithmBasic form: Choose an initial (e.g. Random) weight

vector Let Let If not converged, go back to step 2

For several units: decorrelation Let

Let

wwxwgExwxgEw TT )}({)}({ '

www /

p

jjj

Tppp wwwww

1111

1111 / pTppp wwww

9

Nonlinear ICAModel:

Existence and uniqueness of solutions There always exists an infinity of

solutions if the space of the nonlinear mixing functions is not limited

Post-nonlinear problem

))(()())(()(txhtytsftx

mixing

demixing

f

M

jjijii Mitsaftx

1

,...,1)),(()(

10

Algorithms for Nonlinear ICA

Burel’s approach: neural solution, known nonlinearities on unknown parametersKrob & Benidir: high order moments, polynomial mixturesPajunen et al.: SOMs, locally factorable pdfPajunen et al.: GTM(generative topographic mapping), output distribution matches the known source distributionsPost nonlinear mixtures:

Taleb & Jutten: adaptive componentwise separation Yang et al.: two-layer neural network Puntonet et al.: nonlinearities are a power function,

geometrical considerations