• Home

  • Documents

  • Graphical)Modelsmccallum/courses/gm2011/11...Graphical)Models Lecture)11:) CliqueTrees...
81
Graphical Models Lecture 11: Clique Trees Andrew McCallum [email protected] Thanks to Noah Smith and Carlos Guestrin for some slide materials. 1

Graphical)Modelsmccallum/courses/gm2011/11...Graphical)Models Lecture)11:) CliqueTrees Andrew)McCallum [email protected] Thanks)to)Noah)Smith)and))Carlos)Guestrin)for)some)slide)materials.1

  • Upload
    others

  • View
    5

  • Download
    0

Embed Size (px)

Citation preview

Page 1: Graphical)Modelsmccallum/courses/gm2011/11...Graphical)Models Lecture)11:) CliqueTrees Andrew)McCallum mccallum@cs.umass.edu Thanks)to)Noah)Smith)and))Carlos)Guestrin)for)some)slide)materials.1

Graphical  Models

Lecture  11:    

Clique  TreesAndrew  McCallum

[email protected]

Thanks  to  Noah  Smith  and    Carlos  Guestrin  for  some  slide  materials.1

Page 2: Graphical)Modelsmccallum/courses/gm2011/11...Graphical)Models Lecture)11:) CliqueTrees Andrew)McCallum mccallum@cs.umass.edu Thanks)to)Noah)Smith)and))Carlos)Guestrin)for)some)slide)materials.1

Inference

• Last  two  lectures:    variable  eliminaDon– Sum  out  variables  one  at  a  Dme.

• Today:    alternaDve  algorithms.– Also  based  on  factors.– Central  data  structure:    clique  tree.

Page 3: Graphical)Modelsmccallum/courses/gm2011/11...Graphical)Models Lecture)11:) CliqueTrees Andrew)McCallum mccallum@cs.umass.edu Thanks)to)Noah)Smith)and))Carlos)Guestrin)for)some)slide)materials.1

Formalisms

Bayesian  Network

Markov  Network

pairwise  Markov  Network

factor  graph

moralize  skeleton

triangulateextra  variables  per  factor

nothing

essenDally  equivalent

helpful  for  approximate  inference

clique  tree

helpful  for  exact  inference

triangulatepick  root,  add  direcDons one  

factor  per  clique

Lecture  5

Page 4: Graphical)Modelsmccallum/courses/gm2011/11...Graphical)Models Lecture)11:) CliqueTrees Andrew)McCallum mccallum@cs.umass.edu Thanks)to)Noah)Smith)and))Carlos)Guestrin)for)some)slide)materials.1

Formalisms

Markov  Network

clique  tree

Lecture  5

triangulate

Page 5: Graphical)Modelsmccallum/courses/gm2011/11...Graphical)Models Lecture)11:) CliqueTrees Andrew)McCallum mccallum@cs.umass.edu Thanks)to)Noah)Smith)and))Carlos)Guestrin)for)some)slide)materials.1

Clique  Tree

A

B C

D E

F

ABC

BCD

CDE

DEF

Every  maximal  clique  becomes  a  vertex.

Tree  structure?then  “Clique  Tree”

(If  graph  was  triangulated,  always  get  a  tree  structure.)

Connect  verDceswith  overlappingvariables

Lecture  5

Page 6: Graphical)Modelsmccallum/courses/gm2011/11...Graphical)Models Lecture)11:) CliqueTrees Andrew)McCallum mccallum@cs.umass.edu Thanks)to)Noah)Smith)and))Carlos)Guestrin)for)some)slide)materials.1

Clique  Tree

A

B C

D E

F

ABC

BCD

CDE

DEF

For  each  edge,  intersecDon  of  r.v.s  separates  the  rest  in  H.

sepH(A,  D  |  B,  C)

sepH(B,  E  |  C,  D)

sepH(C,  F  |  D,  E)

Lecture  5

“Sep-­‐set”  =  {A,  D}

Page 7: Graphical)Modelsmccallum/courses/gm2011/11...Graphical)Models Lecture)11:) CliqueTrees Andrew)McCallum mccallum@cs.umass.edu Thanks)to)Noah)Smith)and))Carlos)Guestrin)for)some)slide)materials.1

“Cluster  Graph”• A  cluster  graph  for  a  set  of  factors  Φ  over  X  is  an  undirected  graph,  each  of  whose  nodes  i  is  associated  with  a  subset  Ci  ⊆  X.

• A  cluster  graph  must  be  “family  preserving”:  each  factor  ϕ  ∈  Φ  must  be  associated  with  a  cluster  Ci  (denoted  α(ϕ)),  such  that  Scope[ϕ]  ⊆  Ci.

• Each  edge  between  a  pair  of  clusters  Ci  and  Cj  is  associated  with  a  sepset  Si,j  ⊆  Ci  ∩  Cj.

• “Cluster  Tree”  a  cluster  graph  that  is  a  tree.7

Page 8: Graphical)Modelsmccallum/courses/gm2011/11...Graphical)Models Lecture)11:) CliqueTrees Andrew)McCallum mccallum@cs.umass.edu Thanks)to)Noah)Smith)and))Carlos)Guestrin)for)some)slide)materials.1

“Running  Intersec3on”  Property

• Cluster  tree  T    (with  verDces  Ci…)  has  the  running  intersec3on  property  if,  whenever  there  is  a  variable  X  such  as  X  ∈  Ci  and  X  ∈  Cj,  then  X  is  also  in  every  cluster  in  the  (unique)  path  in  T  between  Ci  and  Cj.

8

ABC BCD CDE

C C C

Page 9: Graphical)Modelsmccallum/courses/gm2011/11...Graphical)Models Lecture)11:) CliqueTrees Andrew)McCallum mccallum@cs.umass.edu Thanks)to)Noah)Smith)and))Carlos)Guestrin)for)some)slide)materials.1

“Clique  Tree”    DefiniDon

• Given  an  undirected  graph  H,  a  tree  T  is  a  clique  tree  for  H if:– Each  node  in  T  corresponds  to  a  clique  in  H

– Each  maximal  clique  in  H is  a  node  in  T

– Each  sepset  Si,j  separates  the  variables  strictly  on  one  side  of  the  edge  from  the  variables  on  the  other  side.

Si,j  =

Ci  ∩  Cj

aka  “Junc0on  Tree,”  or  “Join  Tree”

Clique  Tree  =  cluster  tree  that  sa.sfies  running  intersec.on  property.

Ci Cj

In  other  

words

Page 10: Graphical)Modelsmccallum/courses/gm2011/11...Graphical)Models Lecture)11:) CliqueTrees Andrew)McCallum mccallum@cs.umass.edu Thanks)to)Noah)Smith)and))Carlos)Guestrin)for)some)slide)materials.1

How  to  Obtain  a  Clique  TreeOp,on  1

• Start  with  factor  graph  or  undirected  graph.

• Do  variable  eliminaDon.    On  iteraDon  i:– Let  ψi  be  the  intermediate  factor

– Let  τi  be  the  marginalized  factor

– Let  Ci  be  the  set  of  variables  involved  in  ψi

• One  node  per  Ci,  edge  Ci-­‐Cj  when  τi  gets  used  in  compuDng  ψj.

Page 11: Graphical)Modelsmccallum/courses/gm2011/11...Graphical)Models Lecture)11:) CliqueTrees Andrew)McCallum mccallum@cs.umass.edu Thanks)to)Noah)Smith)and))Carlos)Guestrin)for)some)slide)materials.1

Example

• Eliminate  S. Flu All.

S.I.

R.N. H.

ϕSR ϕSH

ϕAϕF

ϕFAS

Page 12: Graphical)Modelsmccallum/courses/gm2011/11...Graphical)Models Lecture)11:) CliqueTrees Andrew)McCallum mccallum@cs.umass.edu Thanks)to)Noah)Smith)and))Carlos)Guestrin)for)some)slide)materials.1

Example

• Eliminate  S.

• ψ1  =  ϕFAS  ·∙  ϕSR  ·∙  ϕSH

Flu All.

S.I.

R.N. H.

ϕSR ϕSH

ϕAϕF

ϕFAS

FASRH

Page 13: Graphical)Modelsmccallum/courses/gm2011/11...Graphical)Models Lecture)11:) CliqueTrees Andrew)McCallum mccallum@cs.umass.edu Thanks)to)Noah)Smith)and))Carlos)Guestrin)for)some)slide)materials.1

Example

• Eliminate  S.

• ψ1  =  ϕFAS  ·∙  ϕSR  ·∙  ϕSH

Flu All.

S.I.

R.N. H.

ψ1

ϕAϕF

FASRH

Page 14: Graphical)Modelsmccallum/courses/gm2011/11...Graphical)Models Lecture)11:) CliqueTrees Andrew)McCallum mccallum@cs.umass.edu Thanks)to)Noah)Smith)and))Carlos)Guestrin)for)some)slide)materials.1

Example

• Eliminate  S.

• ψ1  =  ϕFAS  ·∙  ϕSR  ·∙  ϕSH

• τ1  =  ∑S  ψ1

Flu All.

S.I.

R.N. H.

ψ1

ϕAϕF

FASRH

Page 15: Graphical)Modelsmccallum/courses/gm2011/11...Graphical)Models Lecture)11:) CliqueTrees Andrew)McCallum mccallum@cs.umass.edu Thanks)to)Noah)Smith)and))Carlos)Guestrin)for)some)slide)materials.1

Example

• Eliminate  S.

• ψ1  =  ϕFAS  ·∙  ϕSR  ·∙  ϕSH

• τ1  =  ∑S  ψ1

Flu All.

R.N. H.

τ1

ϕAϕF

FASRH

Page 16: Graphical)Modelsmccallum/courses/gm2011/11...Graphical)Models Lecture)11:) CliqueTrees Andrew)McCallum mccallum@cs.umass.edu Thanks)to)Noah)Smith)and))Carlos)Guestrin)for)some)slide)materials.1

Example

• Eliminate  R.

• ψ2  =  τ1

• τ2  =  ∑R  ψ2

Flu All.

R.N. H.

τ1

ϕAϕF

FASRH FARH

Page 17: Graphical)Modelsmccallum/courses/gm2011/11...Graphical)Models Lecture)11:) CliqueTrees Andrew)McCallum mccallum@cs.umass.edu Thanks)to)Noah)Smith)and))Carlos)Guestrin)for)some)slide)materials.1

Example

• Eliminate  R.

• ψ2  =  τ1

• τ2  =  ∑R  ψ2

Flu All.

H.

τ2

ϕAϕF

FASRH FARH

Page 18: Graphical)Modelsmccallum/courses/gm2011/11...Graphical)Models Lecture)11:) CliqueTrees Andrew)McCallum mccallum@cs.umass.edu Thanks)to)Noah)Smith)and))Carlos)Guestrin)for)some)slide)materials.1

Example

• Eliminate  H.

• ψ3  =  τ2

• τ3  =  ∑H  ψ3

Flu All.

H.

τ2

ϕAϕF

FASRH FARH FAR

Page 19: Graphical)Modelsmccallum/courses/gm2011/11...Graphical)Models Lecture)11:) CliqueTrees Andrew)McCallum mccallum@cs.umass.edu Thanks)to)Noah)Smith)and))Carlos)Guestrin)for)some)slide)materials.1

Example

• Eliminate  H.

• ψ3  =  τ2

• τ3  =  ∑H  ψ3

Flu All.

τ3

ϕAϕF

FASRH FARH FAR

Page 20: Graphical)Modelsmccallum/courses/gm2011/11...Graphical)Models Lecture)11:) CliqueTrees Andrew)McCallum mccallum@cs.umass.edu Thanks)to)Noah)Smith)and))Carlos)Guestrin)for)some)slide)materials.1

Example

• Eliminate  A.

• ψ4  =  τ3  ϕA

• τ4  =  ∑A  ψ3

Flu All.

τ3

ϕAϕF

FASRH FARH FAR FA

Page 21: Graphical)Modelsmccallum/courses/gm2011/11...Graphical)Models Lecture)11:) CliqueTrees Andrew)McCallum mccallum@cs.umass.edu Thanks)to)Noah)Smith)and))Carlos)Guestrin)for)some)slide)materials.1

Example

• Eliminate  A.

• ψ4  =  τ3  ϕA

• τ4  =  ∑A  ψ3

Flu τ4

ϕF

FASRH FARH FAR FA

Page 22: Graphical)Modelsmccallum/courses/gm2011/11...Graphical)Models Lecture)11:) CliqueTrees Andrew)McCallum mccallum@cs.umass.edu Thanks)to)Noah)Smith)and))Carlos)Guestrin)for)some)slide)materials.1

Example

• Eliminate  F.

• ψ5  =  τ4  ϕF

• τ5  =  ∑F  ψ5

Flu τ4

ϕF

FASRH FARH FAR FA

F

Page 23: Graphical)Modelsmccallum/courses/gm2011/11...Graphical)Models Lecture)11:) CliqueTrees Andrew)McCallum mccallum@cs.umass.edu Thanks)to)Noah)Smith)and))Carlos)Guestrin)for)some)slide)materials.1

Example

• Eliminate  F.

• ψ5  =  τ4  ϕF

• τ5  =  ∑F  ψ5

τ5

FASRH FARH FAR FA

F

Page 24: Graphical)Modelsmccallum/courses/gm2011/11...Graphical)Models Lecture)11:) CliqueTrees Andrew)McCallum mccallum@cs.umass.edu Thanks)to)Noah)Smith)and))Carlos)Guestrin)for)some)slide)materials.1

AlternaDve  Ordering

• {H,  R,  S,  A,  F}

SR

HS

SAF FA F

Flu All.

S.I.

R.N. H.

ϕSR ϕSH

ϕAϕF

ϕFAS

Page 25: Graphical)Modelsmccallum/courses/gm2011/11...Graphical)Models Lecture)11:) CliqueTrees Andrew)McCallum mccallum@cs.umass.edu Thanks)to)Noah)Smith)and))Carlos)Guestrin)for)some)slide)materials.1

How  to  Obtain  a  Clique  TreeOp,on  1

• Start  with  factor  graph  or  undirected  graph.

• Do  variable  eliminaDon.  

• One  node  per  Ci,  edge  Ci-­‐Cj  when  τi  gets  used  in  compuDng  ψj.

• Result  is  always  a  clique  tree!– Running  intersecDon  property.– Different  orderings  will  give  different  clique  trees.

Page 26: Graphical)Modelsmccallum/courses/gm2011/11...Graphical)Models Lecture)11:) CliqueTrees Andrew)McCallum mccallum@cs.umass.edu Thanks)to)Noah)Smith)and))Carlos)Guestrin)for)some)slide)materials.1

Reducing  a  Clique  Tree

• Given  a  clique  tree  T  for  a  set  of  factors  Φ,  there  is  always  a  reduced  clique  tree  T’  such  that– all  clique-­‐verDces  in  T’ are  in  T

– no  clique  vertex  in  T’ is  a  subset  of  another.

• ConstrucDon  exploits  the  running  intersecDon  property.

Page 27: Graphical)Modelsmccallum/courses/gm2011/11...Graphical)Models Lecture)11:) CliqueTrees Andrew)McCallum mccallum@cs.umass.edu Thanks)to)Noah)Smith)and))Carlos)Guestrin)for)some)slide)materials.1

Reduced  Clique  Tree

HS

SR

SAF FA F

Page 28: Graphical)Modelsmccallum/courses/gm2011/11...Graphical)Models Lecture)11:) CliqueTrees Andrew)McCallum mccallum@cs.umass.edu Thanks)to)Noah)Smith)and))Carlos)Guestrin)for)some)slide)materials.1

Reduced  Clique  Tree

HS

SR

SAF FA F

HS

SR

SAF

Page 29: Graphical)Modelsmccallum/courses/gm2011/11...Graphical)Models Lecture)11:) CliqueTrees Andrew)McCallum mccallum@cs.umass.edu Thanks)to)Noah)Smith)and))Carlos)Guestrin)for)some)slide)materials.1

Reduced  Clique  Tree

FASRH FARH FAR FA

F

Page 30: Graphical)Modelsmccallum/courses/gm2011/11...Graphical)Models Lecture)11:) CliqueTrees Andrew)McCallum mccallum@cs.umass.edu Thanks)to)Noah)Smith)and))Carlos)Guestrin)for)some)slide)materials.1

Reduced  Clique  Tree

FASRH FARH FAR FA

F

FASRH

Page 31: Graphical)Modelsmccallum/courses/gm2011/11...Graphical)Models Lecture)11:) CliqueTrees Andrew)McCallum mccallum@cs.umass.edu Thanks)to)Noah)Smith)and))Carlos)Guestrin)for)some)slide)materials.1

How  to  Obtain  a  Clique  TreeOp,on  2

1. Given  factors,  construct  the  undirected  graph.

2. Triangulate  to  get  a  chordal  graph.

3. Find  maximal  cliques  in  the  chordal  graph.

4. Construct  a  tree  from  the  “cluster  graph”  of  maximal  cliques.

Actually  the  method  we  discussed  in  Lecture  5

Page 32: Graphical)Modelsmccallum/courses/gm2011/11...Graphical)Models Lecture)11:) CliqueTrees Andrew)McCallum mccallum@cs.umass.edu Thanks)to)Noah)Smith)and))Carlos)Guestrin)for)some)slide)materials.1

Step  2:  Triangulate

• NP-­‐hard  in  general  to  get  the  smallest  one.

• Use  heurisDcs.

Page 33: Graphical)Modelsmccallum/courses/gm2011/11...Graphical)Models Lecture)11:) CliqueTrees Andrew)McCallum mccallum@cs.umass.edu Thanks)to)Noah)Smith)and))Carlos)Guestrin)for)some)slide)materials.1

Step  3:    Find  Maximal  Cliques

• Maximum  cardinality  search:    gives  an  induced  graph  with  no  fill  edges  (assuming  chordal  graph  as  input,  which  we  have).

• We  proved  last  Dme  that  every  maximal  clique  in  the  induced  graph  equates  to  the  scope  of  an  intermediate  factor  from  VE.

• Other  approaches  exist.

Page 34: Graphical)Modelsmccallum/courses/gm2011/11...Graphical)Models Lecture)11:) CliqueTrees Andrew)McCallum mccallum@cs.umass.edu Thanks)to)Noah)Smith)and))Carlos)Guestrin)for)some)slide)materials.1

Maximum  Cardinality  Searchfor  VE  Ordering

• Start  with  undirected  graph  on  X,  all  nodes  unmarked.

• For  i  =  |X|  to  1:– Let  Y  be  the  unmarked  variable  in  X  with  the  largest  number  of  marked  neighbors

– π(Y)  =  i–Mark  Y.

• Eliminate  using  permutaDon  π.Lecture  10

Page 35: Graphical)Modelsmccallum/courses/gm2011/11...Graphical)Models Lecture)11:) CliqueTrees Andrew)McCallum mccallum@cs.umass.edu Thanks)to)Noah)Smith)and))Carlos)Guestrin)for)some)slide)materials.1

Example

• First  node  is  arbitrary. Flu All.

S.I.

R.N. H.

Page 36: Graphical)Modelsmccallum/courses/gm2011/11...Graphical)Models Lecture)11:) CliqueTrees Andrew)McCallum mccallum@cs.umass.edu Thanks)to)Noah)Smith)and))Carlos)Guestrin)for)some)slide)materials.1

Example

• First  node  is  arbitrary. Flu All.

S.I.

R.N. H.

5

Page 37: Graphical)Modelsmccallum/courses/gm2011/11...Graphical)Models Lecture)11:) CliqueTrees Andrew)McCallum mccallum@cs.umass.edu Thanks)to)Noah)Smith)and))Carlos)Guestrin)for)some)slide)materials.1

Example

• 1:    A,  S

• 0:    R,  H

Flu All.

S.I.

R.N. H.

5

Page 38: Graphical)Modelsmccallum/courses/gm2011/11...Graphical)Models Lecture)11:) CliqueTrees Andrew)McCallum mccallum@cs.umass.edu Thanks)to)Noah)Smith)and))Carlos)Guestrin)for)some)slide)materials.1

Example

• 2:    A

• 1:    R,  H

Flu All.

S.I.

R.N. H.

5

4

Page 39: Graphical)Modelsmccallum/courses/gm2011/11...Graphical)Models Lecture)11:) CliqueTrees Andrew)McCallum mccallum@cs.umass.edu Thanks)to)Noah)Smith)and))Carlos)Guestrin)for)some)slide)materials.1

Example

• 1:    R,  H Flu All.

S.I.

R.N. H.

5

4

3

Page 40: Graphical)Modelsmccallum/courses/gm2011/11...Graphical)Models Lecture)11:) CliqueTrees Andrew)McCallum mccallum@cs.umass.edu Thanks)to)Noah)Smith)and))Carlos)Guestrin)for)some)slide)materials.1

Example

• 1:    H Flu All.

S.I.

R.N. H.

5

4

3

2

Page 41: Graphical)Modelsmccallum/courses/gm2011/11...Graphical)Models Lecture)11:) CliqueTrees Andrew)McCallum mccallum@cs.umass.edu Thanks)to)Noah)Smith)and))Carlos)Guestrin)for)some)slide)materials.1

Example

• Order:    {H,  R,  A,  S,  F} Flu All.

S.I.

R.N. H.

5

4

3

2 1

Page 42: Graphical)Modelsmccallum/courses/gm2011/11...Graphical)Models Lecture)11:) CliqueTrees Andrew)McCallum mccallum@cs.umass.edu Thanks)to)Noah)Smith)and))Carlos)Guestrin)for)some)slide)materials.1

How  to  Obtain  a  Clique  TreeOp,on  2

1. Given  factors,  construct  the  undirected  graph.2. Triangulate  to  get  a  chordal  graph.3. Find  maximal  cliques  in  the  chordal  graph.4. Construct  a  tree  from  the  “cluster  graph”  of  

maximal  cliques.– Results  from  VE  with  the  ordering  from  maximum  

cardinality  search;  or  use  maximum  spanning  tree  (edge  weights  =  sepset  sizes).

Page 43: Graphical)Modelsmccallum/courses/gm2011/11...Graphical)Models Lecture)11:) CliqueTrees Andrew)McCallum mccallum@cs.umass.edu Thanks)to)Noah)Smith)and))Carlos)Guestrin)for)some)slide)materials.1

Example

• Order:    {H,  R,  A,  S,  F} Flu All.

S.I.

R.N. H.

5

4

3

2 1

RS

HS

SF ASF

F

Page 44: Graphical)Modelsmccallum/courses/gm2011/11...Graphical)Models Lecture)11:) CliqueTrees Andrew)McCallum mccallum@cs.umass.edu Thanks)to)Noah)Smith)and))Carlos)Guestrin)for)some)slide)materials.1

Example  (Reduce)

• Order:    {H,  R,  A,  S,  F} Flu All.

S.I.

R.N. H.

5

4

3

2 1

RS

HS

ASF

Page 45: Graphical)Modelsmccallum/courses/gm2011/11...Graphical)Models Lecture)11:) CliqueTrees Andrew)McCallum mccallum@cs.umass.edu Thanks)to)Noah)Smith)and))Carlos)Guestrin)for)some)slide)materials.1

Now  You  Know  …

• How  to  construct  a  clique  tree.

• Also  called  a  “juncDon  tree”  or  a  “join  tree.”

Page 46: Graphical)Modelsmccallum/courses/gm2011/11...Graphical)Models Lecture)11:) CliqueTrees Andrew)McCallum mccallum@cs.umass.edu Thanks)to)Noah)Smith)and))Carlos)Guestrin)for)some)slide)materials.1

Running  IntersecDon  Property

• For  connected  clique-­‐verDces  Ci  and  Cj,  the  sepset  Si,j  is  Ci  ∩  Cj,  and  it  

separates  the  variables  on  the  Ci  side  from  those  on  the  Cj  side.  

RS

HS

ASF FT

Page 47: Graphical)Modelsmccallum/courses/gm2011/11...Graphical)Models Lecture)11:) CliqueTrees Andrew)McCallum mccallum@cs.umass.edu Thanks)to)Noah)Smith)and))Carlos)Guestrin)for)some)slide)materials.1

Running  IntersecDon  Property

A

B C

D E

F

ABC

BCD

CDE

DEF

For  each  edge,  intersecDon  of  r.v.s  separates  the  rest  in  H.

sepH(A,  D  |  B,  C)

sepH(B,  E  |  C,  D)

sepH(C,  F  |  D,  E)

Lecture  5

Page 48: Graphical)Modelsmccallum/courses/gm2011/11...Graphical)Models Lecture)11:) CliqueTrees Andrew)McCallum mccallum@cs.umass.edu Thanks)to)Noah)Smith)and))Carlos)Guestrin)for)some)slide)materials.1

Family  PreservaDon

• Every  factor  in  the  original  graph  is  associated  with  one  clique-­‐vertex  in  the  clique  tree.

Page 49: Graphical)Modelsmccallum/courses/gm2011/11...Graphical)Models Lecture)11:) CliqueTrees Andrew)McCallum mccallum@cs.umass.edu Thanks)to)Noah)Smith)and))Carlos)Guestrin)for)some)slide)materials.1

Family  PreservaDon  Example

HS

SR

SAF FA F

Flu All.

S.I.

R.N. H.

ϕSR ϕSH

ϕAϕF

ϕFAS

ϕSR

ϕSH ϕFAS ϕFϕA

Page 50: Graphical)Modelsmccallum/courses/gm2011/11...Graphical)Models Lecture)11:) CliqueTrees Andrew)McCallum mccallum@cs.umass.edu Thanks)to)Noah)Smith)and))Carlos)Guestrin)for)some)slide)materials.1

Inference

• Last  two  lectures:    variable  eliminaDon– Sum  out  variables  one  at  a  Dme.

• Today:    alternaDve  algorithms.– Also  based  on  factors.– Central  data  structure:    clique  tree.

• What  are  these  algorithms?

Page 51: Graphical)Modelsmccallum/courses/gm2011/11...Graphical)Models Lecture)11:) CliqueTrees Andrew)McCallum mccallum@cs.umass.edu Thanks)to)Noah)Smith)and))Carlos)Guestrin)for)some)slide)materials.1

Clique  Trees  and  Variable  EliminaDon

• Each  product-­‐factor  ψ  belongs  to  a  clique-­‐vertex.  

• Each  τ  is  a  “message”  from  one  clique-­‐vertex  to  another.

Page 52: Graphical)Modelsmccallum/courses/gm2011/11...Graphical)Models Lecture)11:) CliqueTrees Andrew)McCallum mccallum@cs.umass.edu Thanks)to)Noah)Smith)and))Carlos)Guestrin)for)some)slide)materials.1

Example

HS

SR

SAF FA F

Flu All.

S.I.

R.N. H.

ϕSR ϕSH

ϕAϕF

ϕFAS

ϕSR

ϕSH ϕFAS ϕFϕA

• {H,  R,  S,  A,  F}

Page 53: Graphical)Modelsmccallum/courses/gm2011/11...Graphical)Models Lecture)11:) CliqueTrees Andrew)McCallum mccallum@cs.umass.edu Thanks)to)Noah)Smith)and))Carlos)Guestrin)for)some)slide)materials.1

Example

HS

SR

SAF FA F

Flu All.

S.I.

R.N. H.

ϕSR ϕSH

ϕAϕF

ϕFAS

ϕSR

ϕSH ϕFAS ϕFϕA

• {H,  R,  S,  A,  F}

ψSR

ψSH

ψFAS ψFAψF

Just  happens  to  be  that  in  this  graph  with  this  ordering  there  is  a  one-­‐to-­‐one  correspondence  between  original  phis  and  product  factors  psi,  because  there  was  only  one  factor  touching  each  variable-­‐to-­‐eliminate.

Page 54: Graphical)Modelsmccallum/courses/gm2011/11...Graphical)Models Lecture)11:) CliqueTrees Andrew)McCallum mccallum@cs.umass.edu Thanks)to)Noah)Smith)and))Carlos)Guestrin)for)some)slide)materials.1

Messages

• NoDce  that  the  variable  eliminaDon  ordering  implies  a  direcDon  to  the  edges.

• Messages  go  “upstream.”

• This  leads  to  a  clique-­‐tree-­‐centric  view  of  VE,  in  which  the  same  calculaDons  happen  as  in  VE.

Page 55: Graphical)Modelsmccallum/courses/gm2011/11...Graphical)Models Lecture)11:) CliqueTrees Andrew)McCallum mccallum@cs.umass.edu Thanks)to)Noah)Smith)and))Carlos)Guestrin)for)some)slide)materials.1

Example

HS

SR

SAF FA F

Flu All.

S.I.

R.N. H.

ϕSR ϕSH

ϕAϕF

ϕFAS

ϕSR

ϕSH ϕFAS ϕFϕA

• {H,  R,  S,  A,  F}

τS

τS

τAF τF

ψSR

ψSH

ψFAS ψFAψF

Page 56: Graphical)Modelsmccallum/courses/gm2011/11...Graphical)Models Lecture)11:) CliqueTrees Andrew)McCallum mccallum@cs.umass.edu Thanks)to)Noah)Smith)and))Carlos)Guestrin)for)some)slide)materials.1

Example

HS

SR

SAF

Flu All.

S.I.

R.N. H.

ϕSR ϕSH

ϕAϕF

ϕFAS

ϕSR

ϕSH ϕFAS

ϕA

• {H,  R,  S,  A,  F}

• Reduced  version

τS

τS

ψSR

ψSH

ψFAS

ϕF

Page 57: Graphical)Modelsmccallum/courses/gm2011/11...Graphical)Models Lecture)11:) CliqueTrees Andrew)McCallum mccallum@cs.umass.edu Thanks)to)Noah)Smith)and))Carlos)Guestrin)for)some)slide)materials.1

Example

HS

SR

SAF

Flu All.

S.I.

R.N. H.

ϕSR ϕSH

ϕAϕF

ϕFAS

ϕSR

ϕSH νFAS

• {H,  R,  S,  A,  F}

• Reduced  version

• IniDal  potenDals

τS

τS

ψSR

ψSH

ψFAS

νFAS = φFASφAφF

Go  through  math  that  gets  done  for  VE  here.

Page 58: Graphical)Modelsmccallum/courses/gm2011/11...Graphical)Models Lecture)11:) CliqueTrees Andrew)McCallum mccallum@cs.umass.edu Thanks)to)Noah)Smith)and))Carlos)Guestrin)for)some)slide)materials.1

Message  Passing  in  the  Clique  Tree

• Calculate  ini7al  poten7als  for  each  clique-­‐vertex.

• Choose  an  arbitrary  root  clique  Cr.    This  imposes  a  direcDonality  on  the  graph  (think  of  the  root  as  a  sink).

• Pass  messages  “upstream”:

νj(Cj) =�

φ∈Φj

φ

δi→j =�

Ci\Si,j

νi

�

k∈Neighborsi\{j}

δk→i

Ordering  can  be  different  from  the  VE  ordering  that  might  have  created  the  clique  tree.

Page 59: Graphical)Modelsmccallum/courses/gm2011/11...Graphical)Models Lecture)11:) CliqueTrees Andrew)McCallum mccallum@cs.umass.edu Thanks)to)Noah)Smith)and))Carlos)Guestrin)for)some)slide)materials.1

Example

HS

SR

SAF FA F

Flu All.

S.I.

R.N. H.

ϕSR ϕSH

ϕAϕF

ϕFAS

ϕSR

ϕSH ϕFAS ϕFϕA

• {H,  R,  S,  A,  F}

τS’

τS

τAF τF

ψSR

ψSH

ψFAS ψFAψF

Go  through  math  that  gets  done  for  VE  here:EliminaDng  S.Factor  product  of  τS  ,  τS’  and  ϕFAS  to  form  ψFAS.Marginalize  out  S

Page 60: Graphical)Modelsmccallum/courses/gm2011/11...Graphical)Models Lecture)11:) CliqueTrees Andrew)McCallum mccallum@cs.umass.edu Thanks)to)Noah)Smith)and))Carlos)Guestrin)for)some)slide)materials.1

Message  Passing  in  the  Clique  Tree

δi→j =�

Ci\Si,j

νi

�

k∈Neighborsi\{j}

δk→i

incomingmessagesfrom  otherneighbors  of  i

original  factorsinvolving  the  familyof  Ci

sum  out  variables  that  arenot  involved  in  Cj

Page 61: Graphical)Modelsmccallum/courses/gm2011/11...Graphical)Models Lecture)11:) CliqueTrees Andrew)McCallum mccallum@cs.umass.edu Thanks)to)Noah)Smith)and))Carlos)Guestrin)for)some)slide)materials.1

Message  Passing  in  the  Clique  Tree

δi→j =�

Ci\Si,j

νi

�

k∈Neighborsi\{j}

δk→i

ψ

τ

Page 62: Graphical)Modelsmccallum/courses/gm2011/11...Graphical)Models Lecture)11:) CliqueTrees Andrew)McCallum mccallum@cs.umass.edu Thanks)to)Noah)Smith)and))Carlos)Guestrin)for)some)slide)materials.1

Message  Passing  in  the  Clique  Tree

• NoDce  that  we  have  a  recursive  structure  here:    message-­‐sum  of  products  of  message-­‐sums  of  products  of  message-­‐sums  of  …

δi→j =�

Ci\Si,j

νi

�

k∈Neighborsi\{j}

δk→i

Page 63: Graphical)Modelsmccallum/courses/gm2011/11...Graphical)Models Lecture)11:) CliqueTrees Andrew)McCallum mccallum@cs.umass.edu Thanks)to)Noah)Smith)and))Carlos)Guestrin)for)some)slide)materials.1

Message  Passing  in  the  Clique  Tree

• NoDce  that  you  have  to  work  “upward.”

• Root  will  be  last.

δi→j =�

Ci\Si,j

νi

�

k∈Neighborsi\{j}

δk→i

Page 64: Graphical)Modelsmccallum/courses/gm2011/11...Graphical)Models Lecture)11:) CliqueTrees Andrew)McCallum mccallum@cs.umass.edu Thanks)to)Noah)Smith)and))Carlos)Guestrin)for)some)slide)materials.1

Message  Passing  in  the  Clique  Tree

• NoDce  that  you  have  to  work  “upward.”

• Root  r  will  be  last.

βr = νr

�

k∈Neighborsr

δk→r

=�

X\Ci

�

φ∈Φ

φ

= Z · P (Cr)

Product  

of  ini.al

 poten.

als

Messages  

into  roo

t

Unnormalized  marginal  distribuDon  of  root  variable

s  in  Ci

Page 65: Graphical)Modelsmccallum/courses/gm2011/11...Graphical)Models Lecture)11:) CliqueTrees Andrew)McCallum mccallum@cs.umass.edu Thanks)to)Noah)Smith)and))Carlos)Guestrin)for)some)slide)materials.1

The  Algorithm

• Input:    clique  tree  T,  factors  Φ,  root  Cr• For  each  clique  Ci,  calculate  νi  

• While  Cr  is  sDll  waiDng  on  incoming  messages:

– Choose  a  Ci  that  has  received  all  of  its  incoming  messages.

– Calculate  and  send  the  message  from  Ci  to  Cupstream-­‐neighbor(i)

• Return  βr

Page 66: Graphical)Modelsmccallum/courses/gm2011/11...Graphical)Models Lecture)11:) CliqueTrees Andrew)McCallum mccallum@cs.umass.edu Thanks)to)Noah)Smith)and))Carlos)Guestrin)for)some)slide)materials.1

Obtaining  Marginals

• Given  a  set  of  query  variables  Q  that  is  contained  in  some  clique  C:–Make  that  clique  the  root.– Run  the  upward  pass  of  clique  tree  VE,  resulDng  in  factor  β.

– Return  ∑C\Q  β.

• Later:    How  to  get  marginals  for  Q  that  are  not  contained  in  one  clique.

Page 67: Graphical)Modelsmccallum/courses/gm2011/11...Graphical)Models Lecture)11:) CliqueTrees Andrew)McCallum mccallum@cs.umass.edu Thanks)to)Noah)Smith)and))Carlos)Guestrin)for)some)slide)materials.1

Going  Farther

• We  can  modify  this  algorithm  to  give  us  the  marginal  probability  of  every  random  variable  in  the  network.– Naively:    run  clique  tree  VE  with  each  clique  as  the  root.

Page 68: Graphical)Modelsmccallum/courses/gm2011/11...Graphical)Models Lecture)11:) CliqueTrees Andrew)McCallum mccallum@cs.umass.edu Thanks)to)Noah)Smith)and))Carlos)Guestrin)for)some)slide)materials.1

HS

SR

SAF

δHS→SAF

δSR→SAF

root

Page 69: Graphical)Modelsmccallum/courses/gm2011/11...Graphical)Models Lecture)11:) CliqueTrees Andrew)McCallum mccallum@cs.umass.edu Thanks)to)Noah)Smith)and))Carlos)Guestrin)for)some)slide)materials.1

HS

SR

SAF

δSAF→HS

δSR→SAF

root

Same  message  as  needed  for  previous  root!

Page 70: Graphical)Modelsmccallum/courses/gm2011/11...Graphical)Models Lecture)11:) CliqueTrees Andrew)McCallum mccallum@cs.umass.edu Thanks)to)Noah)Smith)and))Carlos)Guestrin)for)some)slide)materials.1

HS

SR

SAF

δHS→SAF

δSAF→SR

root

Page 71: Graphical)Modelsmccallum/courses/gm2011/11...Graphical)Models Lecture)11:) CliqueTrees Andrew)McCallum mccallum@cs.umass.edu Thanks)to)Noah)Smith)and))Carlos)Guestrin)for)some)slide)materials.1

HS

SR

SAF

δHS→SAF

δSR→SAF

root

δSAF→SR

δSAF→HS

Page 72: Graphical)Modelsmccallum/courses/gm2011/11...Graphical)Models Lecture)11:) CliqueTrees Andrew)McCallum mccallum@cs.umass.edu Thanks)to)Noah)Smith)and))Carlos)Guestrin)for)some)slide)materials.1

Going  Farther

• We  can  modify  this  algorithm  to  give  us  the  marginal  probability  of  every  random  variable  in  the  network.– Naively:    run  clique  tree  VE  with  each  clique  as  the  root.

– Be~er:    noDce  that  the  same  messages  get  used  on  different  runs;  only  two  messages  per  clique  tree  edge.

Page 73: Graphical)Modelsmccallum/courses/gm2011/11...Graphical)Models Lecture)11:) CliqueTrees Andrew)McCallum mccallum@cs.umass.edu Thanks)to)Noah)Smith)and))Carlos)Guestrin)for)some)slide)materials.1

Sum-­‐Product  Message  Passing

• Each  clique  tree  vertex  Ci  passes  messages  to  each  of  its  neighbors  once  it’s  ready  to  do  so.– This  is  asynchronous;  might  want  to  be  careful  about  scheduling.

• At  the  end,  for  all  Ci:

– This  is  the  unnormalized  marginal  for  Ci.

βi = νi

�

k∈Neighborsi

δk→i

Page 74: Graphical)Modelsmccallum/courses/gm2011/11...Graphical)Models Lecture)11:) CliqueTrees Andrew)McCallum mccallum@cs.umass.edu Thanks)to)Noah)Smith)and))Carlos)Guestrin)for)some)slide)materials.1

HS

SR

SAF

δHS→SAF

δSR→SAF

root

δSAF→SR

δSAF→HS

βSR

βHS βSAF

Page 75: Graphical)Modelsmccallum/courses/gm2011/11...Graphical)Models Lecture)11:) CliqueTrees Andrew)McCallum mccallum@cs.umass.edu Thanks)to)Noah)Smith)and))Carlos)Guestrin)for)some)slide)materials.1

CalibraDon

• Two  adjacent  cliques  Ci  and  Cj  are  calibrated  when: �

Ci\Si,j

βi =�

Cj\Si,j

βj

Ci CjSi,j  =

Ci  ∩  Cj

βi βj

Page 76: Graphical)Modelsmccallum/courses/gm2011/11...Graphical)Models Lecture)11:) CliqueTrees Andrew)McCallum mccallum@cs.umass.edu Thanks)to)Noah)Smith)and))Carlos)Guestrin)for)some)slide)materials.1

CalibraDon

• Two  adjacent  cliques  Ci  and  Cj  are  calibrated  when:

Ci CjSi,j  =

Ci  ∩  Cj

βi βj

�

Ci\Si,j

βi =�

Cj\Si,j

βj

= µi,j(Si,j)

μi,j

Page 77: Graphical)Modelsmccallum/courses/gm2011/11...Graphical)Models Lecture)11:) CliqueTrees Andrew)McCallum mccallum@cs.umass.edu Thanks)to)Noah)Smith)and))Carlos)Guestrin)for)some)slide)materials.1

Sum-­‐Product  Message  Passing

• Computes  the  marginal  probability  of  all  variables  using  only  twice  the  computaDon  of  the  upward  pass.

• Results  in  a  calibrated  clique  tree.

• A~racDve  if  we  expect  different  kinds  of  queries.

Page 78: Graphical)Modelsmccallum/courses/gm2011/11...Graphical)Models Lecture)11:) CliqueTrees Andrew)McCallum mccallum@cs.umass.edu Thanks)to)Noah)Smith)and))Carlos)Guestrin)for)some)slide)materials.1

Calibrated  Clique  Tree  as  a  Graphical  Model

• Original  (unnormalized)  factor  model  and  calibrated  clique  tree  represent  the  same  (unnormalized)  measure:

�

φ∈Φ

φ =

�

C∈Vertices(T )

βC

�

S∈Edges(T )

µS

unnormalized  GibbsdistribuDon  from  originalfactors  Φ

clique  beliefs

sepset  beliefs

Page 79: Graphical)Modelsmccallum/courses/gm2011/11...Graphical)Models Lecture)11:) CliqueTrees Andrew)McCallum mccallum@cs.umass.edu Thanks)to)Noah)Smith)and))Carlos)Guestrin)for)some)slide)materials.1

Calibrated  Clique  Tree  as  a  Graphical  Model

�

C∈Vertices(T )

βC

�

S∈Edges(T )

µS

=

�

C∈Vertices(T )

νC

�

C�∈Neighbors(C)

δC�→C

�

S=∈Edges(T ):S=C∩C�

µC∩C�

=

�

C∈Vertices(T )

νC

�

C�∈Neighbors(C)

δC�→C

�

S=∈Edges(T ):S=C∩C�

�

C\S

βC

=

�

C∈Vertices(T )

νC

�

C�∈Neighbors(C)

δC�→C

�

S=∈Edges(T ):S=C∩C�

�

C\S

νC

�

C��Neighbors(C)

δC��→C

Page 80: Graphical)Modelsmccallum/courses/gm2011/11...Graphical)Models Lecture)11:) CliqueTrees Andrew)McCallum mccallum@cs.umass.edu Thanks)to)Noah)Smith)and))Carlos)Guestrin)for)some)slide)materials.1

Calibrated  Clique  Tree  as  a  Graphical  Model

µi,j = δCi→Cj δCj→Ci

�

C∈Vertices(T )

βC

�

S∈Edges(T )

µS

= . . .

=

�

C∈Vertices(T )

νC

�

C�∈Neighbors(C)

δC�→C

�

S=∈Edges(T ):S=C∩C�

δC�→C

�

C\S

νC

�

C��∈Neighbors(C)\{C�}

δC��→C

=

�

C∈Vertices(T )

νC

�

C�∈Neighbors(C)

δC�→C

�

S=∈Edges(T ):S=C∩C�

δC�→CδC→C�

=�

C∈Vertices(T )

νC

=�

φ∈Φ

φ

Page 81: Graphical)Modelsmccallum/courses/gm2011/11...Graphical)Models Lecture)11:) CliqueTrees Andrew)McCallum mccallum@cs.umass.edu Thanks)to)Noah)Smith)and))Carlos)Guestrin)for)some)slide)materials.1

What  You  Now  Know

• Can  reinterpret  variable  eliminaDon  as  message  passing  in  a  clique  tree.

• Can  share  computaDon  to  get  lots  of  marginals  with  only  double  the  cost  of  VE.– Sum-­‐product  belief  propagaDon

• Calibrated  clique  tree  is  an  alternaDve  representaDon  of  the  original  Gibbs  distribuDon.


Campbell (McCallum) tunable chanter instructions

Campbell (McCallum) tunable chanter instructions

Documents
Bounding the Probability of Error for High Precision ... · Character Recognition Gary B. Huang GBHUANG@CS.UMASS.EDU Andrew Kae AKAE@CS.UMASS.EDU Department of Computer Science University

Bounding the Probability of Error for High Precision ... · Character Recognition Gary B. Huang [email protected] Andrew Kae [email protected] Department of Computer Science University

Documents
FIERCE By Ian McCallum

FIERCE By Ian McCallum

Education
Toward Unified Graphical Models of Information Extraction and Data Mining Andrew McCallum Computer Science Department University of Massachusetts Amherst

Toward Unified Graphical Models of Information Extraction and Data Mining Andrew McCallum Computer Science Department University of Massachusetts Amherst

Documents
McCallum Graduate School of Business

McCallum Graduate School of Business

Documents
McCallum Ritual & Sexuality Xingu062

McCallum Ritual & Sexuality Xingu062

Documents
BL:A utomaticallyDebugging MemoryLeaksinWebApplicationsBL:A utomaticallyDebugging MemoryLeaksinWebApplications John Vilk University of Massachusetts Amherst, USA jvilk@cs.umass.edu

BL:A utomaticallyDebugging MemoryLeaksinWebApplicationsBL:A utomaticallyDebugging MemoryLeaksinWebApplications John Vilk University of Massachusetts Amherst, USA [email protected]

Documents
McCallum Laytons FMCG Brochure

McCallum Laytons FMCG Brochure

Documents
Bennett T. McCallum Monetary Economics

Bennett T. McCallum Monetary Economics

Documents
Radiation Protection Legislation Stephen McCallum

Radiation Protection Legislation Stephen McCallum

Documents
Rosie McCallum and Ana Margarida Sardo

Rosie McCallum and Ana Margarida Sardo

Documents
Learning LINKS - McCallum Theatre

Learning LINKS - McCallum Theatre

Documents
Rajarshi Das rajarshi@cs.umass.edu University of

Rajarshi Das [email protected] University of

Documents
McCallum Layton's Analytics Brochure

McCallum Layton's Analytics Brochure

Documents
International Monetary Economics by Bennett McCallum

International Monetary Economics by Bennett McCallum

Documents
Phil1 1Dennis McCallum

Phil1 1Dennis McCallum

Documents
McCallum Bagpipes Black Acetyl Bagpipes Brochure

McCallum Bagpipes Black Acetyl Bagpipes Brochure

Documents
Bennett T. McCallum Carnegie Mellon University - …economics.uwo.ca/conference/monetaryeconomics05/McCallum.pdf · Bennett T. McCallum Carnegie Mellon University ... receive commodity

Bennett T. McCallum Carnegie Mellon University - …economics.uwo.ca/conference/monetaryeconomics05/McCallum.pdf · Bennett T. McCallum Carnegie Mellon University ... receive commodity

Documents
McCallum Highlandwear Highland Dress Brochure

McCallum Highlandwear Highland Dress Brochure

Documents
Matt McCallum CV

Matt McCallum CV

Career
Casino Royale Invitation - McCallum Theatre

Casino Royale Invitation - McCallum Theatre

Documents
1 An Introduction to Conditional Random Fields for ...mccallum/papers/crf-tutorial.pdf1 An Introduction to Conditional Random Fields for Relational Learning ... A graphical model is

1 An Introduction to Conditional Random Fields for ...mccallum/papers/crf-tutorial.pdf1 An Introduction to Conditional Random Fields for Relational Learning ... A graphical model is

Documents
Bayden McCallum

Bayden McCallum

Documents
Matt McCallum Full Profile

Matt McCallum Full Profile

Documents
Eden McCallum UK Consumer Survey

Eden McCallum UK Consumer Survey

Documents
Open Shortest Path First Protocol (OSPF) Sudarshan Vasudevan svasu@cs.umass.edu

Open Shortest Path First Protocol (OSPF) Sudarshan Vasudevan [email protected]

Documents
May 2018 @ McCallum Theatre

May 2018 @ McCallum Theatre

Documents
Breathe - 26 March 2017 - Bruce McCallum

Breathe - 26 March 2017 - Bruce McCallum

Spiritual
Text Classification with Limited Labeled Data Andrew McCallum mccallum@cs.cmu.edu Just Research (formerly JPRC) Center for Automated Learning and Discovery,

Text Classification with Limited Labeled Data Andrew McCallum [email protected] Just Research (formerly JPRC) Center for Automated Learning and Discovery,

Documents
10 AM CIRQUE - McCallum Theatre

10 AM CIRQUE - McCallum Theatre

Documents