Upload
others
View
1
Download
0
Embed Size (px)
Citation preview
BIT
SF4
64:M
achi
neLe
arni
ng
22Gra
phical
Model
:Baye
sianBelie
fNetwork
sD
r.K
amle
shTi
war
iA
ssis
tant
Pro
fess
or,D
epar
tmen
tofC
SIS
,B
ITS
Pila
ni,P
ilani
Cam
pus,
Raj
asth
an-3
3303
1IN
DIA
Mar
ch15
,202
1O
NLI
NE
(Cam
pus
@B
ITS
-Pila
niJa
n-M
ay20
21)
http://ktiwari.in/ml
Bay
esia
nLe
arni
ng
Add
ress
esm
ostp
roba
ble
clas
sific
atio
nof
new
inst
ance
inst
ead
ofbe
sthy
poth
esis
ford
ata.
Sea
rchi
nga
poss
ibili
tyto
dobe
ttert
hen
MA
PB
ayes
optim
alcl
assi
ficat
ion:
argmax
v j∈V
�h i∈H
P(v
j|hi)
P(h
i|D)
Out
perfo
rms
onan
aver
age
but,
quite
cost
lyto
appl
yG
IBB
SA
lgor
ithm
:C
hoos
ea
hypo
thes
ish∈
Hat
rand
om,
acco
rdin
gto
the
post
erio
rpro
babi
lity
dist
ribut
ion
over
H.
(Exp
ecte
dm
iscl
assi
ficat
ion
erro
ris
boun
ded
toth
etw
ice
ofth
eB
ayes
optim
alcl
assi
fier)
Nai
veB
ayes
Cla
ssifi
er:
assu
mes
inde
pend
ence
give
nth
eta
rget
valu
eargmax
v j∈V
P(a
1,a 2
,...,a
n|v
j)P(v
j)
argmax
v j∈V
P(v
j)Π
iP(a
i|vj)
Hig
hly
prac
tical
met
hod
Mac
hine
Lear
ning
(BIT
SF4
64)
MW
F(1
0-11
AM
)onl
ine@
BIT
S-P
ilani
Lect
ure-
22(M
arch
15,2
021)
2/1
6
Pro
babi
lity
P(x,y
)=
P(x)×
P(y|x)
Inde
pend
ence
ofx
and
yim
plie
sP(y|x)=
P(y)
Then
P(x,y
)=
P(x)×
P(y)
Bay
esR
ule
P(x|y)=
P(x,y
)
P(y)
=P(y|x)×
P(x)
P(y)
Mar
gina
l:di
strib
utio
nof
asi
ngle
varia
ble
xca
nbe
obta
ined
from
agi
ven
join
tdis
trib
utio
np(
x,y)
by
p(x)
=� y
p(x,
y)
The
proc
ess
ofco
mpu
ting
am
argi
nalf
rom
ajo
intd
istr
ibut
ion
isca
lled
mar
gina
lisat
ion.
p(x 1,...,x
i−1,
x i+
1,...,
x n)=
� x i
p(x 1,x
2,...,
x n)
Mac
hine
Lear
ning
(BIT
SF4
64)
MW
F(1
0-11
AM
)onl
ine@
BIT
S-P
ilani
Lect
ure-
22(M
arch
15,2
021)
3/1
6
Mar
gina
lisat
ion
Con
ditio
nalI
ndep
ende
nce
whe
ntw
ova
riabl
ear
ein
depe
nden
tof
each
othe
r,pr
ovid
edth
atw
ekn
owst
ate
ofso
me
othe
rvar
iabl
e
P(x,y
|z)=
P(x|z)×
P(y|z)
Con
side
r:C
asso
ftX
OR
AB
p(C
=1|A,B
)
00
0.10
01
0.99
10
0.80
11
0.25
Ifp(
A=1
)=
0.65
,p(B
=1)=
0.77
Det
erm
ine
p(A
=1|C
=0)
p(A
=1,
C=0
)=
�B
p(A
=1,
B,
C=
0)
=� B
p(C
=0|
A=
1,B)p(A
=1)
p(B)
=p(
C=
0|A
=1,
B=
0)p(
A=
1)p(
B=
0)
+p(
C=
0|A
=1,
B=
1)p(
A=
1)p(
B=
1)
=0.
2×
0.65
×0.
23+
0.75
×0.
65×
0.77
=0.
405
Sim
ilarly
p(A
=0,
C=0
)=
0.07
5
p(A
=1|C
=0)=
p(A=
1,C=
0)p(
C=
0)=
p(A=
1,C=
0)p(
A=
1,C=
0)+
p(A=
0,C=
0)
=0.
843
Mac
hine
Lear
ning
(BIT
SF4
64)
MW
F(1
0-11
AM
)onl
ine@
BIT
S-P
ilani
Lect
ure-
22(M
arch
15,2
021)
4/1
6
Let’s
see
this
Sta
tem
ent:
� J
� p(J|
R)×
f(R)� =
f(R)
Pro
of:
� J
� p(J|
R)×
f(R)�
=� J
� p(J,R
)
p(R)
×f(
R)�
=
�J� p(
J,R)×
f(R)�
p(R)
=f(
R)×
�J
p(J,
R)
p(R)
=f(
R)×
p(R)
p(R)
=f(
R)
Mac
hine
Lear
ning
(BIT
SF4
64)
MW
F(1
0-11
AM
)onl
ine@
BIT
S-P
ilani
Lect
ure-
22(M
arch
15,2
021)
5/1
6
Bel
iefn
etw
ork
(agr
aphi
calm
odel
)
Bel
iefn
etw
ork
uses
grap
hsto
repr
esen
tind
epen
denc
eam
ong
the
varia
bles
inpr
obab
ilist
icm
odel
Inde
pend
ently
spec
ifyin
gal
lthe
attr
ibut
edis
over
kill
With
dist
ribut
ion
ofn
attr
ibut
es,m
argi
nalf
oron
eta
kes
O(2
n−1 )
By
cons
train
ing
varia
ble
inte
ract
ion
(spe
cify
ing
inde
pend
ence
)on
eca
nge
tthe
form
like
p(x 1,x
2,...,
x 100)=
Π99 i=
1φ(x
i,x i+
1)
Bel
iefn
etw
orks
are
aco
nven
ient
fram
ewor
kfo
rrep
rese
ntin
gsu
chin
depe
nden
ceas
sum
ptio
nsB
elie
fnet
wor
ksar
eal
soca
lled
asB
ayes
’Net
wor
ksor
Bay
esia
nB
elie
fNet
wor
ks
Mac
hine
Lear
ning
(BIT
SF4
64)
MW
F(1
0-11
AM
)onl
ine@
BIT
S-P
ilani
Lect
ure-
22(M
arch
15,2
021)
6/1
6
Mod
elin
gIn
depe
nden
cies
One
mor
ning
Trac
eyle
aves
herh
ouse
and
real
ities
that
herg
rass
isw
et.
Isit
due
toov
erni
ghtr
ain
ordi
dsh
efo
rget
totu
rnof
fthe
sprin
kler
last
nigh
t?N
exts
heno
tices
that
the
gras
sof
hern
eigh
bor,
Jack
,is
also
wet
.
(R=1
)→ra
inla
stni
ght,
(S=1
)→sp
rinkl
eron
last
nigh
t,(J
=1)→
Jack
’sgr
ass
isw
et,
(T=1
)→Tr
acey
a’s
Gra
ssis
wet
Mod
elof
Trac
eya’
sw
orld
invo
lves
prob
abili
tydi
strib
utio
non
T,J
,R,S
that
has
24=
16st
ates
Mac
hine
Lear
ning
(BIT
SF4
64)
MW
F(1
0-11
AM
)onl
ine@
BIT
S-P
ilani
Lect
ure-
22(M
arch
15,2
021)
7/1
6
Mod
elin
gIn
depe
nden
cies
(con
td..)
How
ever
we
know
p(T,J
,R,S
)=
p(T|J,R
,S)p(J,R
,S)
=p(
T|J,R
,S)p(J|R
,S)p(R
,S)
=p(
T|J,R
,S)p(J|R
,S)p(R
|S)p(S
)
Com
puta
tion
ofp(
T|J,R
,S)
requ
ires
usto
spec
ify23
=8
valu
esW
ithp(
T=
1|J,
R,S
),on
eca
nus
eno
rmal
izat
ion
toco
mpu
tep(
T=
0|J,
R,S
)as
1−
p(T
=1|
J,R,S
)
Com
puta
tion
ofot
herf
acto
rsw
ould
also
need
4+2+
1va
lues
Tota
lwe
need
8+4+
2+1=
15va
lues
Mac
hine
Lear
ning
(BIT
SF4
64)
MW
F(1
0-11
AM
)onl
ine@
BIT
S-P
ilani
Lect
ure-
22(M
arch
15,2
021)
8/1
6
Con
ditio
nalI
ndep
ende
nce
We
may
assu
me
that
Trac
eya’
sgr
ass
isw
etde
pend
son
lydi
rect
lyon
whe
ther
orno
tith
asbe
enra
inin
gan
dw
heth
eror
noth
ersp
rinkl
erw
ason
sop(
T|J,R
,S)=
p(T|R
,S)
Ass
ume
that
Jack
’sgr
ass
isw
etis
influ
ence
don
lydi
rect
lyby
whe
ther
orno
tith
asbe
enra
inin
gp(
J|R,S
)=
p(J|
R)
Furt
herm
ore,
we
assu
me
the
rain
isno
tdire
ctly
influ
ence
dby
the
sprin
kler
p(R|S)=
p(R)
Ther
efor
e,ou
rmod
elbe
com
es
p(T,J
,R,S
)=
p(T|J,R
,S)p(J|R
,S)p(R
|S)p(S
)
=p(
T|R
,S)p(J|R
)p(R
)p(S
)
Num
bero
fval
ues
we
need
tosp
ecify
is4+
2+1+
1=8
We
can
repr
esen
tthe
seco
nditi
onal
inde
pend
ence
as
Mac
hine
Lear
ning
(BIT
SF4
64)
MW
F(1
0-11
AM
)onl
ine@
BIT
S-P
ilani
Lect
ure-
22(M
arch
15,2
021)
9/1
6
Bel
iefn
etw
ork
Ans
wer
s“h
owto
repr
esen
tthe
seco
nditi
onal
inde
pend
ence
?”B
elie
fnet
wor
kis
adi
strib
utio
nof
the
form
p(x 1,x
2,...,
x n)=
Πn i=
1p(x
i|pa(
x i))
whe
repa
(xi)
repr
esen
tthe
pare
ntal
varia
bles
ofva
riabl
ex i
Rep
rese
nted
asa
dire
cted
grap
h,w
ithan
arro
wpo
intin
gfro
ma
pare
ntva
riabl
eto
child
varia
ble,
abe
liefn
etw
ork
corr
espo
nds
toa
Dire
cted
Acy
clic
Gra
ph(D
AG
)
Mac
hine
Lear
ning
(BIT
SF4
64)
MW
F(1
0-11
AM
)onl
ine@
BIT
S-P
ilani
Lect
ure-
22(M
arch
15,2
021)
10/1
6
Exa
mpl
eO
nem
orni
ngTr
acey
real
ises
that
herg
rass
isw
etan
dth
egr
ass
ofhe
rne
ighb
our,
Jack
,is
also
wet
.Le
tthe
prio
rpro
babi
litie
sbe
p(R
=1)=
0.2
and
p(S
=1)=
0.1.
We
setp
(J=1
|R=1
)=
1,p(
J=1|
R=0
)=
0.2,
p(T
=1|R
=1,S
=0)=
1,p(
T=1
|R=1
,S=1
)=
1,p(
T=1
|R=0
,S=1
)=
0.9,
p(T
=1|R
=0,S
=0)=
0
Usi
ngfo
llow
ing
Bel
eifN
etw
ork;
calc
ulat
e
1P
roba
bilit
yth
atsp
rinkl
erw
asO
Nov
erni
ght,
give
nth
atTr
acey
a’s
gras
sis
wet
.p(
S=1
|T=1
)=
0.33
82H
ow?
onne
xt
slid
e
2P
roba
bilit
yth
atsp
rinkl
erw
asO
Nov
erni
ght,
give
nth
atTr
acey
a’s
gras
sis
wet
and
Jack
’sgr
ass
isal
sow
et.
p(S
=1|T
=1,J
=1)
=0.
1604
Mac
hine
Lear
ning
(BIT
SF4
64)
MW
F(1
0-11
AM
)onl
ine@
BIT
S-P
ilani
Lect
ure-
22(M
arch
15,2
021)
11/1
6
Exa
mpl
e:P
roba
bilit
yof
p(S
=1|T
=1)
p(S=
1|T
=1)
=p(
S=
1,T
=1)
p(T
=1)
=
�J,
Rp(
S=
1,J,
R,T
=1)
�J,
R,S
p(T
=1,
J,R,S
)(1
)
=
�J,
Rp(
J|R)p(T
=1|
R,S
=1)
p(R)p(S
=1)
�J,
R,S
p(J|
R)p(T
=1|
R,S
)p(R
)p(S
)
=
�R
p(T
=1|
R,S
=1)
p(R)p(S
=1)
�R,S
p(T
=1|
R,S
)p(R
)p(S
)(2
)
=0.
9×
0.8×
0.1+
1×
0.2×
0.1
.9×.8
×.1
+1×.2
×.1
+0×.8
×.9
+1×.2
×.9
=0.
3382
Use
sgi
ven
belie
fnet
wor
kin
(1)a
ndpr
oofi
n(2
)
Mac
hine
Lear
ning
(BIT
SF4
64)
MW
F(1
0-11
AM
)onl
ine@
BIT
S-P
ilani
Lect
ure-
22(M
arch
15,2
021)
12/1
6
Unc
erta
inev
iden
ceS
ofto
run
cert
ain
evid
ence
,let
dom(x)=
{red
,blu
e,gr
een}
and
the
vect
ory=
(0.6,0
.1,0
.3)
repr
esen
tsth
ebe
liefi
nth
ere
spec
tive
stat
es.
Har
dev
iden
cear
elik
e(0
,0,1
).A
ssum
ptio
nis
that
p(x|
y,y)
=p(
x|y)
p(x|
y)=
�y
p(x,
y|y)
=�
yp(
x|y,
y)p(
y|y)
=�
yp(
x|y)
p(y|
y)w
here
p(y=
i|y)
repr
esen
tsth
epr
obab
ility
that
yis
inst
ate
iD
ashe
dci
rcle
isus
edto
repr
esen
tava
riabl
ein
soft-
evid
ence
stat
e
Exa
mpl
e:Le
tpro
babi
lity
offir
e,w
hen
ther
eis
afir
eal
arm
is0.
9.M
onu
said
heis
70%
confi
dent
that
heha
dhe
ard
afir
eal
arm
.W
hati
spr
obab
ility
offir
e.
p(F
=1|
A)=
�A
p(F
=1|
A)p(A
|A)=
p(F
=1|
A=
0)p(
A=
0|A)+
p(F
=1|
A=
1)p(
A=
1|A)=
0.1×
0.3+
0.9×
0.7=
0.66
Mac
hine
Lear
ning
(BIT
SF4
64)
MW
F(1
0-11
AM
)onl
ine@
BIT
S-P
ilani
Lect
ure-
22(M
arch
15,2
021)
13/1
6
Unr
elia
ble
Evi
denc
e
Unr
elia
ble:
(con
tinue
dfro
mpr
evio
usex
ampl
e)Ia
sked
Raj
uab
outt
heal
arm
.It
isbe
lieve
dth
atif
alar
mha
dso
und,
ther
eis
80%
chan
ceth
atR
aju
wou
ldte
llit
soun
d.If
alar
mha
dN
OT
soun
ded,
ther
eis
70%
chan
ceth
athe
wou
ldte
llN
OT
soun
d.U
nrel
iabl
eev
iden
ces
are
mod
eled
usin
gda
shed
lines
asbe
low
Mac
hine
Lear
ning
(BIT
SF4
64)
MW
F(1
0-11
AM
)onl
ine@
BIT
S-P
ilani
Lect
ure-
22(M
arch
15,2
021)
14/1
6
Exa
mpl
e
LetB
=1,m
eans
Hol
mes
hous
eha
sbe
enbu
rgle
d.A
=1,m
eans
alar
mw
ento
ff.W
=1,m
eans
Wat
son
hear
dal
arm
.G
=1,m
eans
Gib
bon
hear
dal
arm
.
(a)B
Nfo
rthe
envi
ronm
ent,
(b)I
fGib
bon
isa
little
defa
ndis
only
80%
sure
abou
tthe
alar
mso
und
bein
ghe
ard,
(c)R
epla
cem
ento
fevi
denc
e,(d
)Hol
mes
feel
sW
atso
n’s
obse
rvat
ion
isun
relia
ble
Mac
hine
Lear
ning
(BIT
SF4
64)
MW
F(1
0-11
AM
)onl
ine@
BIT
S-P
ilani
Lect
ure-
22(M
arch
15,2
021)
15/1
6
Than
kYo
u!
Than
kyo
uve
rym
uch
for
your
atte
ntio
n!
Que
ries
?
(Ref
eren
ce1)
1[1
]Tex
tBoo
k:ch
:1/2
/3B
ayes
ian
Rea
soni
ngan
dM
achi
neLe
arni
ng,b
yD
avid
Bar
ber
Mac
hine
Lear
ning
(BIT
SF4
64)
MW
F(1
0-11
AM
)onl
ine@
BIT
S-P
ilani
Lect
ure-
22(M
arch
15,2
021)
16/1
6