10 December, 200810 December, 2008 CIMCA2008 (Vienna)CIMCA2008 (Vienna) 11
Statistical Inferences by Gaussian Markov Statistical Inferences by Gaussian Markov Random Fields on Complex NetworksRandom Fields on Complex Networks
Kazuyuki Tanaka, Kazuyuki Tanaka, TakafumiTakafumi UsuiUsui,,MunekiMuneki Yasuda Yasuda
Graduate School of Information Sciences,Graduate School of Information Sciences,Tohoku UniversityTohoku University
10 December, 200810 December, 2008 CIMCA2008 (Vienna)CIMCA2008 (Vienna) 22
Bayesian Network and Bayesian Network and Graphical modelGraphical model
Image Processing
Regular Graph
Code TheoryRandom Graph
Bipartite Graph
Complete Graph
Data Mining
Machine Learning
Probabilistic Inference
Hypergraph
Bayesian networks are Bayesian networks are formulated for formulated for statistical inferences as statistical inferences as probabilistic models on probabilistic models on various networks. various networks. The performances The performances sometimes depend on sometimes depend on the statistical the statistical properties in the properties in the network structures.network structures.
10 December, 2008 CIMCA2008 (Vienna) 3
Recently, various kinds of networks and their stochastically generating models are interested in the
applications of statistical inferences.
Complex Networks
id i vertex of degree the:
∑∈
∝Vi
diddP ,)( δ
Degree Distribution
1
23
4
5{1,2}{2,3}
{3,4}{2,4}
{3,5}
11 =d 15 =d33 =d
24 =d32 =d
)(dP
1 2 3 4 50 d
:Set of all the vertices}5,4,3,2,1{=V{ }}5,3{},4,3{},4,2{},3,2{},2,1{=E :Set of all the edges
The degree of each vertex plays an important role for the statistical properties in the structures of networks.Networks are classified by using the degree distributions.
(The degree of vertex is the number of edges connected to the vertex)
10 December, 2008 CIMCA2008 (Vienna) 4
Degree Distribution in Complex Networks
γ−ddP ~)(
Poisson Distribution
Power Law DistributionScale Free Network:
Random Network:
0
0.1
0.2
0.3
0.4
0.5
0.6
0 20 40 60 80 100 120
P(k)
k
scale-freerandom
d
P(d)
Random Network
Scale Free Network
It is known that the degree distributions of random networks areaccording to the Poisson distributions. The scale free networks have some hub-vertices, their degree distributions are given by power law distributions.
d
ddP ρ
!1~)(
10-4
10-3
10-2
10-1
100
100 101 102 103P(
k)
k
scale-freerandom
dP(
d)
10 December, 2008 CIMCA2008 (Vienna) 5
Purpose of the present talk
In the present paper, we analyze the statistical performance of the Bayesian inferences on some complex networks including scale free networks. We adopt the Gauss Markov random field model as a probabilistic model in statistical inferences.The statistical quantities for the Gauss Markov random field model can be calculated by using the multi-dimensional Gaussian integral formulas.
10 December, 2008 CIMCA2008 (Vienna) 6
Prior Probability in Bayesian Inference
( )
( )⎟⎠⎞
⎜⎝⎛ +−
+=
⎟⎟⎠
⎞⎜⎜⎝
⎛−−−= ∑∑
∈∈
xCIxCI
xxxZ
xP
V
Vii
Ejiji
rr
r
)(21exp
)2(det
21)(
21exp
),(1,
T||
2
},{
2
αβπ
αβ
βαβα
βα
),( +∞−∞∈∀ ix ⎟⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜⎜
⎝
⎛
=
5
4
3
2
1
xxxxx
xr1
23
4
5{1,2}{2,3}
{3,4}{2,4}
{3,5}
11 =d 15 =d33 =d
24 =d32 =d
⎪⎩
⎪⎨
⎧∈−
==
otherwise0},{1 Eji
jidjCi
i
We adopt the Gauss Markov random field model as a prior probability of Bayesian statistics and the source signals are assumed to be generated by according to the prior probability.
I: Unit Matrix
10 December, 2008 CIMCA2008 (Vienna) 7
Data Generating Process in Bayesian Statistical Inference
( ) ( )∏∈
⎟⎠⎞
⎜⎝⎛ −−=
Viii yxxyP 2
22 21exp
2
1,σπσ
σrr
1
23
4
5{1,2}{2,3}
{3,4}
{2,4}
{3,5}
1
2
3
4
5
xi yi
Additive White Gaussian Noise
As data generating processes, we assume that the observed data are generated from the source signals by adding the white Gaussian noise.
10 December, 2008 CIMCA2008 (Vienna) 8
( ) ( )
yCII
I
dxdxdx,yxPxyh V
r
Lrrr
Lrr
)(
,,,,,
2
||21
αβσ
σβασβα
++=
= ∫ ∫ ∫+∞∞−
+∞∞−
+∞∞−
Bayesian Statistics
( ) ( ) ( )( )σβα
βασσβα
,,,,
,,,yP
xPxyPyxP r
rrrrr
=
xr g
( )βα ,xP r ( )σ,xyP rr yrSource Signal Data
Prior Probability Density Function
Posterior Probability Density Function
Data Generating Process
10 December, 2008 CIMCA2008 (Vienna) 9
Prior ProbabilityDensityFunction
Statistical Performance by Sample Average
( )xP r
)1(xr
)2(xr
)3(xr
10 December, 2008 CIMCA2008 (Vienna) 10
Prior ProbabilityDensityFunction
Statistical Performance by Sample Average
( )xP r
)1,1(y
)1(xr
)2(xr
)3(xr
)2,1(yr
)1,2(yr
)2,2(yr
)1,3(yr
)2,3(yrData Generating Process
( ))2(xyP r
( ))1(xyP rr
( ))3(xyP rr
10 December, 2008 CIMCA2008 (Vienna) 11
Prior ProbabilityDensityFunction
Posterior Probability Density Function
Statistical Performance by Sample Average
( )xP r
)1,1(y
)1(xr
)2(xr
)3(xr
)2,1(yr
)1,2(yr
)2,2(yr
)1,3(yr
)2,3(yrData Generating Process
( ))2(xyP r
( ))1(xyP rr
( ))3(xyP rr
)1,1(hr
)2,1(hr
)1,2(hr
)2,2(hr
)1,3(hr
)2,3(hr
( ))1,1(yxP rr
( ))2,1(yxP rr
( ))1,2(yxP rr
( ))2,2(yxP rr
( ))1,3(yxP rr
( ))2,3(yxP rr
10 December, 2008 CIMCA2008 (Vienna) 12
Prior ProbabilityDensityFunction
Posterior Probability Density Function
Statistical Performance by Sample Average
( ) ∑ ∑= =
−≡3
1
2
1
2)(),(
||61,,
k lkxlkh
VE rr
σβα
( )xP r
)1,1(y
)1(xr
)2(xr
)3(xr
)2,1(yr
)1,2(yr
)2,2(yr
)1,3(yr
)2,3(yrData Generating Process
( ))2(xyP r
( ))1(xyP rr
( ))3(xyP rr
)1,1(hr
)2,1(hr
)1,2(hr
)2,2(hr
)1,3(hr
)2,3(hr
( ))1,1(yxP rr
( ))2,1(yxP rr
( ))1,2(yxP rr
( ))2,2(yxP rr
( ))1,3(yxP rr
( ))2,3(yxP rr
10 December, 2008 CIMCA2008 (Vienna) 13
Statistical Performance Analysis
( ) ( ) ( )
( ) ( ) ( )∫ ∫
∫ ∫
−=
−≡
ydxdxPxyPxyhV
ydxdyxPxyhV
E
rrrrrrrr
rrrrrrr
βασσβα
σβασβασβα
,,,,,1
,,,,,,1,,
2
2
( )σβα ,,,yh rr
g
( )βα ,xP r yrPrior Probability Density Function
Data Generating Process
( ),σxyP rrxr
Posterior Probability Density Function
( )σ,yxP ,,βαrr
10 December, 2008 CIMCA2008 (Vienna) 14
The exact expression of the average for the mean square error with respect to the source signals and the observable data can be derived.
Statistical Performance Analysis
( ) ( ) ( ) ( )
)(Tr1
,,,,,1,,
2
2
2
CIII
V
xdydxPxyPxyhV
E
αβσσ
βασσβασβα
++=
−= ∫ ∫rrrrrrrr
( ) ( )⎟⎠⎞
⎜⎝⎛ +−
+= xCIxCIxP V
rrr )(21exp
)2(det, T
|| αβπ
αββα
( ) ( )∏∈
⎟⎠⎞
⎜⎝⎛ −−=
Viii yxxyP 2
221exp
21,
σσπσrr
( )( )( )⎪
⎩
⎪⎨
⎧∈−
==
otherwise0},{1 Eji
jidjCi
i
( )
yCII
Iyh
r
rr
)(
,,,
2 αβσ
σβα
++=
Data Generating Process
Prior Probability Density Function
10 December, 2008 CIMCA2008 (Vienna) 15
Erdos and Renyi (ER) modelThe following procedures are repeated:• Choose a pair of vertices {i, j} randomly.• Add a new edge and connect to the selected
vertices if the pair of vertices have no edge.
0.0
0.1
0.2
0.3
0.4
0.5
0 5 10 15 20
P(k
)k0 5 10 15 20
0
P(d) Poisson Distribution
d
0.5
Random Network
10 December, 2008 CIMCA2008 (Vienna) 16
Barabasi and Albert (BA) modelThe following procedures are repeated:• Choose a vertex i with the probability which
is proportional to the degree of vertex i.• Add a new vertex with an edge and connect
to the selected vertices.
Scale Free Network
21
21
1)2(1 =X 1)2(1 =X
2)3(1 =X 1)3(2 =X
1)3(3 =X41
41
42
10-5
10-4
10-3
10-2
10-1
100
100 101 102 103
P(k
)
k
P(d)
d
10 December, 2008 CIMCA2008 (Vienna) 17
Assign a fitness parameter μ(i) to each vertex i using the uniform distribution on the interval [0, 1].
Ohkubo and Horiguchi (OH) modelThe following procedures are repeated:• Select an edge {i, j} randomly.• Select a vertex k preferentially with the
probability that is proportional to (dk + 1)μ(k)
• Rewire the edge {i, j} to {i,k} if {i,k} is not edge.
Scale Free Network
i
j
k ki
j
10 December, 2008 CIMCA2008 (Vienna) 18
Statistical Performance for GMRF model on Complex Networks
Random Network by ER model
Scale Free Network by OH model
Scale Free Network by BA model
Remove all the isolated vertices
10 December, 200810 December, 2008 CIMCA2008 (Vienna)CIMCA2008 (Vienna) 1919
SummarySummary
Statistical Performance of Probabilistic Statistical Performance of Probabilistic Inference by Gauss Markov Random field Inference by Gauss Markov Random field models has been derived for various models has been derived for various complex networks.complex networks.We have given some numerical calculations We have given some numerical calculations of statistical performances for various of statistical performances for various complex networks including Scale Free complex networks including Scale Free Networks as well as Random Networks. Networks as well as Random Networks.