G lobal O ptimality of the S uccessive M ax B et A lgorithm

Global Optimality of the Successive MaxBet Algorithm

USC ENITIAA de NANTES

FranceMohamed HANAFI

and

Jos M.F. TEN BERGE

Department of psychology University of Groningen

The Netherlands

Global Optimality of the Successive MaxBet Algorithm

Summary. 1. The Successive MaxBet Problem (SMP). 2. The MaxBet Algorithm. 3. Global Optimality : Motivation/Problems. 4. Conclusions and Open questions.

1. The Successive MaxBet Problem (S.M.P)

jk pp ,kjA

kjAA Kjk ,,2,1,

KK , Blocks Matrix

s.p.s.d

KKKK

K

K

AAA

AAA

AAA

A

21

22221

11211

Auuu '

order 1

K

jkjkjk

1,

' uAu

Maximize

Subject to

11

''

K

kkkuuuu

1. The Successive MaxBet Problem (S.M.P)

Kk ,,2,1 1' kkuu

order s

Kk ,,2,1

Auuuuu '21 ,...,, K

Maximize

Subject to1' kkuu

0uU kk'{

121 skkkk uuuU Kk ,,2,1

1. The Successive MaxBet Problem (S.M.P)

2. The Successive MaxBet AlgorithmTen Berge (1986,1988)

Order 1

K

jjkjk

1

uAv

1. Take arbitrary initial unit length vectors ku

2. Compute :

3. rescale vk to unit length, and set uk= vk

4. Repeat steps 2 and 3 till convergence

Kk ,,2,1

Kk ,,2,1

Kk ,,2,1

Order s

2. The Successive MaxBet AlgorithmTen Berge (1986,1988)

''jjpkjkkpkj jk

UUIAUUIA

K

jjkjk

1

uAv

1. Take arbitrary initial unit length vectors ku

2. Compute :

3. rescale vk to unit length, and set uk= vk

4. Repeat steps 2 and 3 till convergence

Kk ,,2,1

Kk ,,2,1

Kk ,,2,1

Property 1 : Convergence of the MaxBet Algorithm

u

u

Property 2 : Necessary Condition of Convergence

1' kkuu

Kk ,,2,1

KKKmmmm

m

m

u

u

u

u

u

u

AAA

AAA

AAA

22

11

2

1

21

22221

11211

K ....21u

3. Motivation and results

1. MaxBet Algorithm depends on the starting vector

2. MaxBet algorithm does not guarantee the computation of the global solution of SMP

43 23 -13 0 -7

23 31 10 1 0

-13 10 64 -19 -2

0 1 -19 24 18

-7 0 -2 18 58

A

11A 12A

22A21A

3. Motivation and results : an example

2K 21 p 32 p

42185 64023

(u)= 10621Function value{ 3846.7 5978.4

(v)= 9825.1

0.67 0.36 0.20 0.53 0.30

{Starting Vector *u

0.64 0.31 0.64 0.24 0.10

*v

0.69 0.72 0.58 -0.43 -0.68

v{Solution Vector

0.94 0.31 -0.92 0.35 0.11

u

3. Motivation and results: Two Questions

Q1. How can we know that the solution computed by the Maxbet algorithm is global or not ?

Q2. When the solution is not global, how can we reach using this solution the global solution ?

K

K

pKKKKK

Kp

Kp

IAAA

AIAA

AAIA

A

21

222221

112111

,...,,2

1

21

3. Motivation and Results : Proceeding

Global solution of SMP

Spectral properties (eigenvalues and eigenvectors) of K ,...,, 21

A

When, for a solution, {u, 1,

2, …,

K} satisfies

KKKKKKK

K

K

u

u

u

u

u

u

AAA

AAA

AAA

22

11

2

1

21

22221

11211

RESULT 1

then u is the global solution of SMP.

is negative semidefinite, K ,...,, 21A

Result 1

we have :vAv= vAv 1v1

v1

2v2v2

… KvK

vK

= (v) 1

2 …

K,

hence(v)1+2+…+K= (u)

the matrix A is negative semidefinite,

ELEMENTS OF PROOF (Result 1)

To what extent the previous sufficient condition (Result 1):

is necessary ?

K ,...,, 21A(matrix is negative semi definite)

3. Motivation and Results

matrix blocks 2,2 is A

RESULT 2

22

11

2

1

2221

1211

u

u

u

u

AA

AA

When u is the global maximum of S.M.P it verifies :

2

1

2122221

12111

,p

p

IAA

AIAA

21 ,Athen matrix is negative semi definite

Result 2

Suppose has a positive eigenvalue 21 ,A

021 , wwA

1. w is block-normed vector

2. w is not block-normed vector

2.1. w is not block orthogonal to u

2.2. w is block orthogonal to u

ELEMENTS OF PROOF (Result 2)

1. w is block-normed vector

021 ,

' wAw

uAww 21'

w is better solution than u

2. w is not block-normed vector2.1. w is not block orthogonal to u

222

22211

222112

)'(8)''(

)''(

wuwwww

wwww

222

22211

2222

)'(8)''(

)'(16

wuwwww

wu

wuv

v is better solution than u

*wuv

v is better solution than u

211

122

)'(

)'(

dud

dudq

0'

0'

22

11

ud

ud

qww t*

w is not block-normed vector2.2. w is block orthogonal to u

RESULT 2

Result 3

positive are elements allwith

matrix blocks , is KKA

K

kkpp

1

ghaA phg ....,2,1,

then matrix is negative semi definite K ,...,, 21A

When u is the global maximum of S.M.P

positive are elements allwith

matrix blocks , is KKA

Suppose has a positive eigenvalue

0

K ,...,, 21A

wwA K,...,, 21

ELEMENTS OF PROOF (Result 3)

p

hgghhg

p

hgghhg auuauuu

,,

||

u has all elements of the same sign

ELEMENTS OF PROOF (Result 3)

k

K

kkkK

1

'',...,,

'

21wwAwwwAw

w has all elements of the same sign

2K

sign same theof elements allnot with

matrix blocks , is KKA

The sufficient condition (Result 1) :

is not necessary

K ,...,, 21A(matrix is negative semi definite)

Result 4

45 -20 5 6 16 3

-20 77 -20 -25 -8 -21

5 -20 74 47 18 -32

6 -25 47 54 7 -11

16 -8 18 7 21 -7

3 -21 -32 -11 -7 70

ELEMENTS OF PROOF (Result 4)

A

3K

21 p

22 p

23 p

0.49-0.87 0.80 0.59 0.56-0.82

(u) =378.96

Random research with 10.000.000 starting vectors

=0.48 u =

ELEMENTS OF PROOF (Result 4)

- Possible Application in statistics :

Multivariate Methods (Analysis of K sets of data )

4. General Conclusions

1. Generalized canonical correlationAnalysis: Horst (1961)

3. Soft Modeling Approach :Estimation of latent variables under mode B

Wold (1984); Hanafi (2001)

2. Rotation methods : MaxDiff, MaxBet, generalized Procrustes Analysis

Gower(1975); Van de Geer(1984);Ten Berge (1986,1988)

-

- Necessary condition for the case K=3 when matrix A has not all

elements of the same sign?

4. Perspective and Little Open Question

2X

0

0

2'

1'

aXa

aXa

nn,

??0a

nn,1X

uf

Ku

u

u

u2

11k

'kuu

K,...,,k 21

Motivation: Illustration 1 MaxBet Algorithm depends on the starting vector

The Successive MaxBet Problem (S.M.P)and

Multivariate Methods

kpn, K,,,k 21kX

K

kkpn

1

, KXXXX 21

Some multivarite methods

Generalized canonical correlation methods

Rotation methods(Agreement methods)

SOFT MODELING APPRAOCH(Approch)

n

XXA

'

kjAA

Rotation methods

njk

kj

XXA

'

S M P = MaxBet method Van de Geer (1984) Ten Berge (1986,1988)

Kjk ,,2,1,

S M P = MaxBet method Van de Geer (1984) Ten Berge (1986,1988)

Kjk ,,2,1,

S M P = MaxDiff method Van de Geer (1984) Ten Berge (1986,1988)

Kjk ,,2,1,

jk

jkn

jk

kj

0

' XXA

S M P = MaxBet method Van de Geer (1984) Ten Berge (1986,1988)

Kjk ,,2,1, Kjk ,,2,1,

jkn

K

jkn

jk

jk

kj XX

XX

A '

'

2

S M P = Generalized Procrustes Analysis Gower(1975), Ten Berge (1986,1988)

kX 'kkkk WPX

Generalized canonical correlation methods

SVD

kjAA n

jkkj

PPA

'

SMP = Horst method(1961)

kjAA njk

kjkj

PPA

'

S M P = Soft Modeling Appraoch (Hanafi 2001)

1,1,0 jkkj

Mode B soft modeling approach

uuAuuuIAu ''' mm mK Auu '

uIAu m' MaximizeK,...,,k 211 Subject to k

'kuu

1k'kuu K,...,,k 21

Auu ' MaximizeK,...,,k 211 Subject to k

'kuu

Auuuuu '21 ,...,, K

Kk ,,2,1 1' kkuu

Maximize

Subject to

KKKmmmm

m

m

u

u

u

u

u

u

AAA

AAA

AAA

22

11

2

1

21

22221

11211

0u

Multivariate Eigenvalue Problem Watterson and Chu(1993)

kppp ....2 solutions ofnumber 21

1' kkuuKk ,,2,1

KKKmmmm

m

m

u

u

u

u

u

u

AAA

AAA

AAA

22

11

2

1

21

22221

11211