Illustration of Shannon's Theorem
Embed Size (px)
Citation preview
-
8/10/2019 Illustration of Shannon's Theorem
1/15
pL (1p)L
p
p
L
-
8/10/2019 Illustration of Shannon's Theorem
2/15
B() L
p
(1 p)
B()
0 20 40 60 80 100
0.0
0.2
0.4
0.6
0.8
1.0
p = 0.2 (1 p) = 0.8
L
pL
(1 p)L
pL (1 p)L
-
8/10/2019 Illustration of Shannon's Theorem
3/15
np(m)
m0 1000 2000 3000 4000
0
5
10
15
20
L= 5000 p
L m L
m/L= p
B()
L
(1 p)log2 11p+ p log2 1p
L
2L
HB(p) = (1 p)log21
1 p + p log21
p,
B()
m L m
np(m) = L!
m!(L m)! (1 p)Lmpm,
p
m!
2mm
e
m,
np(m)
np(m) L
2m(L
m) L
e L
e(1 p)L
m
Lm
ep
mm
,
-
8/10/2019 Illustration of Shannon's Theorem
4/15
np(m)/2L
m
0 200 400 600 800 1000
0.01
0.03
L = 1000 p m L
p L
Lp(1 p) m/L
mL =p
np(m)
1
2LmL(1 mL)
1 p1 mL
L(1mL)
pmL
LmL
.
m
L =q,
np(q) = np(m) 12Lq(1 q)
1 p1 q
L(1q)
pq
Lq
,
logen
p(q) logenp(m) = loge1
2Lq(1 q)+ L [(1 q)(loge(1 p) loge(1 q)) +q(loge(p) loge(q))]
.
Lp
Lp(1 p) q p+
np() = logen
p(q) 12loge[2L(p+)(1 p )]+ L [(1 p ) (loge(1 p) loge(1 p )) +
(p+) (loge(p) loge(p+))] ,
np() 1
2loge(2Lp(1 p)) +
2(1 p)+
2p
+ L
(1 p )
1 p (p+)
p
.
L
np()
1
2
loge(2Lp(1
p))
L
2
2p(1 p).
-
8/10/2019 Illustration of Shannon's Theorem
5/15
log[np(m)]
m
0 200 400 600 800 1000
-40
-30
-20
-10
0
L = 1000 p
mL
1
2loge
L
2p(1 p)LmL p
22p(1 p) .
L L(p ) L(p+) L m
m
m/L0.15 0.20 0.25 0.30 0.35
30
70
L= 1000 L= 5000
p
mL
p > s
-
8/10/2019 Illustration of Shannon's Theorem
6/15
m
m/L0.0 0.2 0.4 0.6 0.8 1.0
-30
-20
-10
0
10
L= 1000 L= 5000
p
L2p(1p)
pL
(1 p)L
pL
pL
(1 p)L n1/2(p) =
L!
(pL)![(1 p)L]! ,
2L
L
pL=0 n1/2(p) =L
pL=0L!
(pL)![(1
p)L]!
= 2L.
n1/2(p) =
2L
Le
L
2LpLpe
Lp2L(1 p)
L(1p)
e
L(1p) .
n1/2(p) = 1
2Lp(1 p)pLp(1 p)L(1p),
n1/2(p) = 1
2Lp(1 p)2L(p log2p+(1p) log2(1p)) ,
n1/2(p) = 2L(p log2p+(1p) log2(1p))
12 log2(2Lp(1p)).
L
LHB(p) = L (p log2p (1 p)log2(1 p)),
n1/2(p) = 2LHB(p).
Lp
2L(HB(p)+)
L(HB
(p) +)
-
8/10/2019 Illustration of Shannon's Theorem
7/15
-
8/10/2019 Illustration of Shannon's Theorem
8/15
n1/2(x) n1/2(p)e2L[xp](x1+p) < n1/2(p)e2(12p)L[xp].
Lp
Lx=0
n1/2(x) LHB ,
M
L
-
8/10/2019 Illustration of Shannon's Theorem
11/15
L Lp 2M
2M 2L ML
Lp
Lp
2M 2L
(2L)!(2LM)!
2L
2LM2L2
L(LM)2LM
e2L2LM .
2L
2M
2L
2L 2M L M
M L
p
2M
Lp
Lp
L
p L
2L(1HB(p)) 2M 2LHB M
2M2LHB 2L.
p= 0, HB(p) = 0 2L
p= 12
HB(p) = 1
2LLHB(p) 2LHB(p) 2L
-
8/10/2019 Illustration of Shannon's Theorem
12/15
L 2M L2 2L
n1/2(x) = L!
(xL)![(1 x)L]!.
L2
2M (2M1)
(2M 1)2LHB 2M L(p+)
Lp
Q=
Lp
xL=01
2Ln1/2(x) =
Lp
xL=0L!
xL!(L xL)!
1
2L
,
Q=
LpxL=0
12L(1 x)x
1
(1 x)L(1x)1
(x)Lx
1
2L
.
Q n1/2(p)2L
,
Q
2LHB
2L
.
2M 1 Lp 2M Lp
Lx=012L
n1/2(x)
p 1L 0.0 0.2 0.4 0.6 0.8 1.0
-150
-100
-50
L= 5000
L= 1000
L
2M L ML
(a b)
(a) + (b);
-
8/10/2019 Illustration of Shannon's Theorem
13/15
S=
2M11
Q 2MQ 2M+LHBL.
M+ LHB L
-
8/10/2019 Illustration of Shannon's Theorem
14/15
-
8/10/2019 Illustration of Shannon's Theorem
15/15
