Learning the structure of Deep sparse Graphical Model

Ryan Prescott Adams Hanna M Wallach

Zoubin Ghahramani

Presented by Zhengming Xing

Some pictures are directly copied from the paper and Hanna Wallach’s slides

outline

• Introduction

• Finite belief network

• Infinite belief network

• Inference

• Experiment

Introduction Main contribution: combine deep belief network and nonparametric bayesian together.

Main idea: use IBP to learn the structure of the network

Structure of the network include:

Depth

Width

Connectivity

Single layer networkUse Binary matrix to represent the network.

Black refer to 1(two unit were connected)

White refer to 0 (two unit were not connected)

IBP can be used as the prior for infinite columns binary matrix

Z

Review IBP

)(poisson

)1/( nnk

1.First customer tries dishes.

2. Nth customer tries

Tasked dishes K with probability

new dishes))1/(( npoisson

1/ nnk

Multi-layer network

Cascading IBP),( Also parameterize by

Each dishes in the restaurant is also a customer in another Indian buffet process

Each matrix is exchangeable both rows and columns

This chain can reach the state with probability one ( number of unit in layer m)

Properties:

For unit in layer m+1

Expected number of parents:

Expected number of children:

0)( mK

K

k kK

1 1/

)(mK

Sample from the CIBP prior

model)()1(11 )( mmmmm ZWy m refer to the layers and increase upto M.

1)1)/(exp(2(.)

),0(~)( )()()()()(

x

Ny mk

mk

mk

mk

mk

weights bias

Place layer wise Gaussian prior on weights and bias, Gamma prior on noise precision

Inference

Weights, bias, noise variance can be sampled with Gibbs sampler.

Inference( sample Z)Two step:

1.

2.

Sample existing dishes

MH-sample

Add a new unit and, and insert connection to this unit with

For a exist unit remove the connection to this unit with

MH ratio

MH ratio

Experiment result

Olivetti faces

Remove bottom halves of the test image.

Experiment result

MNIST Digits

Experiment resultFrey Faces