Towards Independent Subspace Analysis in Controlled

Towards Independent Subspace Analysis inControlled Dynamical Systems

Zoltán Szabó, András Lorincz

Neural Information Processing Group,Department of Information Systems,

Eötvös Loránd University,Budapest, Hungary

ICA Research Network 2008

Acknowledgements:

Zoltán Szabó, András Lorincz nipg.inf.elte.hu

Tools to Integrate-1 (Independent Subspace Analysis)

Cocktail party problemGeneralization of ICA:

multidimensional components,groups of ’people/music bands’

Hidden, independent, multidimensional processes – NOCONTROL.


Tools to Integrate-2 (D-optimal Identification ofDynamical Systems)

Problem: estimate the parameters of a fully observablecontrolled dynamical system by the ‘optimal’ choice of thecontrol.

’Parameters’: dynamics, noise.’Optimal’: in information theoretical sense → D-optimality.

Synonyms: active learning, optimal experimental design.

For ARX models: QP in Bayesian framework.

Controlled dynamical system – FULLY OBSERVABLE.


Motivation

Goal: integrate the former methodologieshidden multidimensional sources,optimal design in controlled systems.


Motivation


EEG data analysis: recognition + prediction, e.g., epileptic


Motivation


EEG data analysis: recognition + prediction, e.g., epileptic

EU-FP7: interested in partnership


Contents

1 Independent Subspace Analysis2 D-optimal ARX Identification

︸︷︷︸

↓

3 D-optimal Hidden ARX Identification4 Illustrations


Independent Subspace Analysis (ISA/MICA)

ISA equations: Observation x is linear mixture ofindependent multidimensional components:

x(t) = As(t),

s(t) = [s1(t); . . . ; sM(t)],

wheresm(t) ∈ R

dm are i.i.d. sampled random variables in time,I(s1, . . . , sM) = 0,mixing matrix A ∈ R

D×D is invertible, with D := dim(s).

Goal: s. Specially for ∀dm = 1: ICA.

Ambiguities: permutation, linear (/orth.) transformation.


D-optimal ARX Identification

Observation equation (ARX model; u: control, e: noise):

s(t + 1) = Fs(t) + Bu(t + 1) + e(t + 1).

Task: ’efficient’ estimation ofsystem parameters: Θ = [F, B, parameters(e)], ornoise: e

by the ’optimal’ choice of control u.

Optimality (D-optimal/’InfoMax’):

Jpar (ut+1) := I(Θ, st+1|st , st−1, . . . , ut+1, ut , . . .) → maxut+1∈U

, or

Jnoise(ut+1) := I(et+1, st+1|st , st−1, . . . , ut+1, ut , . . .) → maxut+1∈U

.

Result (Póczos & Lorincz, 2008): In the Bayesian setting,optimization of J can be reduced to QP.


D-optimal Hidden ARX Identification

State (s) + observation (x) equation:

s(t + 1) = Fs(t) + Bu(t + 1) + e(t + 1), (1)

x(t) = As(t). (2)

Assumptions: I(e1, . . . , eM) = 0 – hidden non-Gaussianindependent multiD componentsTrick: reduce the problem to the fully observable case(d-dep. CLT) + ISA:

x(t + 1) = [AFA−1]x(t) + [AB]u(t + 1) + [Ae(t + 1)],x – ’fully observable tool’ → [AFA−1], [AB], Ae,Ae – ISA→ A} ⇒ F, B, e.

Note: for higher order ARX systems the same idea holds.


Databases, Performance Measure, Questions

Databases (3D-geom, ABC, celebrities):

Performance measure: Amari-index (r ) ∈ [0, 1], 0–perfect.Questions:

1 Dependence on δu = |Ucontrol|,2 Dependence on J = deg(Bcontrol[z]),3 Dependencies on I = deg(FAR[z]) and λ:

FAR[z] =

I−1∏

i=0

(I − λOiz) (|λ| < 1, λ ∈ R, Oi : RND orth.).


Illustrations: Dependence on δu = |Ucontrol|

Decline of the estimation error: power-law [r(T ) ∝ T−c (c > 0)]

1 2 5 10 20 50

10−2

10−1

100

3D−geom (I=3, J=3, λ=0.95)

Number of samples (T)

Am

ari−

inde

x (r

)

δu:0.01

δu:0.1

δu:0.2

δu:0.5

δu:1

x1031 2 5 10 20 50

10−2

10−1

100

I=3, J=3, δu=0.2, λ=0.95

Number of samples (T)A

mar

i−in

dex

(r)

3D−geomABCcelebrities

x103


Illustration: 3D-geom

observation:



observation:

estimated innovation (input of ISA):



observation:

estimated innovation (input of ISA):

Hinton-diagram, estimated components:


Illustration: Dependencies on J = deg(Bcontrol[z]),I = deg(FAR[z])

Precise even for J = 50; I = 50 (ր, λ ց)

1 2 5 10 20 50

10−2

10−1

100

3D−geom (I=3, δu=0.2, λ=0.95)


Am

ari−

inde

x (r

)

J:3J:5J:10J:20J:50

x1031 2 5 10 20 50

10−2

10−1

100

3D−geom (J=1, δu=0.2)


Am

ari−

inde

x (r

)

I:5, λ:0.8I:5, λ:0.85I:5, λ:0.9I:10, λ:0.6I:10, λ:0.65I:10, λ:0.7I:50, λ:0.25I:50, λ:0.3

x103


Summary

Integration of two methodologies:hidden independent multidimensional sources,optimal design in controlled dynamical systems.

Numerical experiences:Decline of the estimation error follows power-law.Robust against:

the order of the AR process,temporal memory of the control.

Possibility to apply ICA, ISA, . . . in controlled systems.


TYFYA!


References

B. Póczos and A. Lorincz.

D-optimal Bayesian interrogation for parameter and noise identificationof recurrent neural networks.

2008.

(submitted; available at http://arxiv.org/abs/0801.1883).

V. V. Petrov.

Central limit theorem for m-dependent variables.

In Proceedings of the All-Union Conference on Probability Theory andMathematical Statistics, pages 38–44, 1958.

Z. Szabó.

Separation Principles in Independent Process Analysis.PhD thesis, Eötvös Loránd University, Budapest, 2008.

(submitted; available at nipg.inf.elte.hu: Download).


http://arxiv.org/abs/0801.1883

nipg.inf.elte.hu

Documents

Towards Independent Subspace Analysis in Controlled