View
0
Download
0
Category
Preview:
Citation preview
Mean Field Asymptotic in Statistical Learning . Apr 12
Lecture 21 .
Derivation of AMP .
I.
① From Belief propagation to message passing algorithm
wrong intuition : the beliefs are approximately Gaussian .
Intuition : In the update rule , only the mean and vom of input beliefs are important .
Input beliefs can be approximated by Gaussian disc .
in the update rule .
Real belief is still non- Gaussian.
Def ( Message passing algorithm) mean and variance of beliefs.
{ Misa ,Vita
,mita.si ,
Jhansi }aeF,iev
,WoE IR
. Messages .
ktl
Update rule : Calculating { misa .viii. niiiii
,Tia } using { mki-a.vika.miii.ii.si } .
I- E
pika Cx ; ) = Fi exp f- }
.
density of N(mita , Vika)→a
fat, exit =#exp f - C×izF÷÷ }.
density of Nlniaki , i'Isi)21T Takei
ri'alxi) & 4iki) Telma Fb it" ) Non- backtracking
÷÷:÷÷:÷÷i÷÷÷÷:÷÷÷÷÷:÷:::÷÷÷i:÷÷¥¥÷÷Extract marginals :
rikki ) & 4- Giblets ; f iGi )
( Pita ,Taka. } input beliefs
,
assumed to be Gaussian
{ rita ,Taji } output beliefs , possibly non - Gaussian
② Example : LASSO with finite temperature p
k
⑦ ri 's:'(Xi ) a expl - Bakit } x exp { - E,a7÷ }a expf - p talxil ) ) .
Pls 'iIa5'= Eat 's.it '
:÷÷÷÷÷:÷÷÷÷÷÷÷÷.¥:÷:c.Vitti = Varxinaep.a.oi.in
, s . (Xi )
where ICP .x. 0.5 ) d. exp { - PC 1- Nxt ) )
.
⑦ r:*. .sn:"::i÷÷÷÷÷÷%This is a Gaussian integration , which gives a Gaussian density .
a exp C - } .2J ktla→i
Aai-ma→i=Ya-¥iAajm'K
Aut - Iasi = ¥. Aaj'
Y'Eat f-
④ Simplification as p → • .
( This step is unnecessary if D= finite)
'ping Ex - acs.ao.si ( x )
= arg;nin{ + Nxt}= 710 ; d 3) Soft thresholding .
hi:b B. Varxnxip.x.gs, [ X] = f 5 ,if 101 > as
,
o, if 101 Cds
.
= S - n' ( O 's RS ).
We are short of alphabets . Abuse notation :
Mita ← him. Mika .
nia.
-
← him. niaki .
Vika ← times. P - Vita,
Tati← limo P - vats..
.
(Sita)" = TEA X 's.it '
(Si'
Oita = Ffa Fist' ni 's.si
mittal = 71 Oita ; dSita)÷÷÷÷:i÷i:÷i÷÷:÷÷z r k
Aai - Va-si= ¥. Aaj
'
Y'Ia t l
r k k
change of variable : zE→ ;= Aai -
ma → i= Ya - Fi Aaj mj→a
Takes ; E Aki - Jka.si = Fei Aaj'
Vjkatl
Assume Aai niid Unit ( { Ifn } ).
Can replace Aai by ht .
K Fa Abi Zb ( t k⇒ Oi→a =boilE t / 2b i
i EV
beta a EF
initial = moi'saids a) there are
ZE-si = Ya - 7¥. Aaj m a2x nxd messages,,±=µ⇐,,⇒viii
'
= Sit.
- n' ( Oita ,xsiia )
.
Takei = In ¥. YES . + I
③ From message passing to approximate message passing .
a÷÷÷:÷i..:÷:::::÷¥:::A crude approximation : throw away non-backtracking property .
O ,"=FAbizbk.lt#
k it iz
-1nF 'Hb mii.my/miiiaa a
mill = M ( Oik ; RSF ) miiiaiffmiisiZE = Ya - F Aaj mjks:::i÷÷:÷÷÷÷siTk = In E ujk + I ⇒ Tk = Lhjvjktla j
simplifying this givesT
mktl = M ( Mkt Azk ; ask) End.::.::÷÷÷i÷÷÷i::
Doesn't give a correct approx .
Needs a correction .
The equations for S,V and T are fine
.
The equations for O , m , Z are off by OG ) .
Derivation of AMP .
④ Replace Vika by Uk, Sita by Sk
.
Oi a = TEA Abi Zb ⑥
:÷÷::i÷÷÷÷i:÷3kt ' = skxtn.IQ, y
' ( Mkt Azk ; ask) ; t I
④ Lee Oita = Oi't S Oita
Misa = Mik t Smita
Zaki ; = Zak t 8 2- a'Ii
⑨ : Oita = F Abi Zbti - Aai Zaks i--
OF SO ;-1
Oik = F Abi Zbii = F. Abi Zbkt BE Abiszbi
{ SO.
= - Aai Za -5. = - Aai ( Zaktszasi ) K - Aai Zak
⑤ : Miki'= 7 ( Oita ; ask) x y Loi
'
; ask) -127 coikixsklsoika-
fmi"
= qq.is , ask ,
mi"" Smita'
Smita = an ( oik ; ask )SOiEa .
② : Zaks ; = Ya - F Aby' Mj# a t Haim a
--
zka SZa
{ Zak = Ya - jZAbjmj→ka= Ya - F.Abjmjk - F. Abjsmjta
S Zita = Aai mia.= Aai (Mit Smita ) re Aai Mik .
⇒ no::÷÷::÷÷÷÷÷÷*nii*iZak = ya - F Abjmjk t F. Abj 27 ( Oj"
; d 3k ) Zaki
= Ya - F Abj Mj t th F. 27 lojk ; a 3k) Zak ,
⇒ ::÷::::÷:÷÷:n.⇒3kt ' = Skx In Iq, y' ( Oj
"
; ask) t 1
This is a version of the AMP for LASSO
Recommended