New Information Inequalities in Terms of One Parametric

1

Journal of mathematics and computer science 15 (2015), 1-22

New Information Inequalities in Terms of One Parametric Generalized

Divergence Measure and Application

K. C. Jain1,*, Praphull Chhabra1,+

1 Department of Mathematics, Malaviya National Institute of Technology, Jaipur (Rajasthan), India

*[email protected]

[email protected], [email protected]

Article history:

Received December 2014

Accepted January 2015

Available online January 2015

Abstract In this work, firstly we introduce the new information divergence measure, characterize it and get the

mathematical relations with other divergences. Further, we introduce new information inequalities on the

new generalized f- divergence measure in terms of the well-known one parametric generalized

divergence. Further, we obtain bounds of the new divergence and the Relative J- divergence as an

application of new information inequalities by using Logarithmic power mean and Identric mean, together

with numerical verification by taking two discrete probability distributions: Binomial and Poisson.

Approximate relations of the new divergence and Relative J- divergence with Chi- square divergence,

have been obtained respectively.

Index terms: New divergence; New information inequalities; Parametric generalized divergence;

Bounds; Logarithmic power mean; Identric mean; Binomial and Poisson distributions; Asymptotic

approximation.

Mathematics Subject Classification: Primary 94A17, Secondary 26D15.

1. Introduction

Divergence measures are basically measures of distance between two probability distributions or compare

two probability distributions, i.e., divergence measures are directly propositional to the distance between

two probability distributions. It means that any divergence measure must take its minimum value zero

when probability distributions are equal and maximum when probability distributions are perpendicular to

each other. So, any divergence measure must increase as probability distributions move apart. Depending

on the nature of the problem, different divergence measures are suitable. So it is always desirable to

develop a new divergence measure.

K.C. Jain, P. Chhabra / J. Math. Computer Sci. 15 (2015), 1-22

2

Divergence measures have been demonstrated very useful in a variety of disciplines such as economics

and political science [34, 35], biology [26], analysis of contingency tables [14], approximation of

probability distributions [8, 23], signal processing [21, 22], pattern recognition [2, 7, 20], color image

segmentation [24], 3D image segmentation and word alignment [33], cost- sensitive classification for

medical diagnosis [28], magnetic resonance image analysis [36] etc.

Also we can use divergences in fuzzy mathematics as fuzzy directed divergences and fuzzy entropies [1,

15, 19], which are very useful to find the amount of average ambiguity or difficulty in making a decision

whether an element belongs to a set or not. Fuzzy information measures have recently found applications

to fuzzy aircraft control, fuzzy traffic control, engineering, medicines, computer science, management and

decision making etc.

Without essential loss of insight, we have restricted ourselves to discrete probability distributions, so let

1 2 3

1

, , ..., : 0, 1 , 2n

n n i i

i

P p p p p p p n

be the set of all complete finite discrete

probability distributions. The restriction here to discrete distributions is only for convenience, similar

results hold for continuous distributions. If we take 0ip for some 1, 2, 3,...,i n , then we have to

suppose that 0

0 0 0 00

f f

.

Some generalized functional information divergence measures had been introduced, characterized and

applied in variety of fields, such as: Csiszar’s f - divergence [9, 10], Bregman’s f - divergence [5],

Burbea- Rao’s f - divergence [6], Renyi’s like f - divergence [27] etc. Similarly, Jain and Saraswat [18]

defined new generalized f - divergence measure, which is given by

1

,2

ni i

f i

i i

p qS P Q q f

q

, (1.1)

where : 0,f R (set of real no.) is real, continuous, and convex function and

1 2 3 1 2 3, , ..., , , , ...,n n nP p p p p Q q q q q , where ip and iq are probability mass functions.

The advantage of these generalized divergences is that many divergence measures can be obtained from

these generalized f - measures by suitably defining the function f . Some resultant divergences by

,fS P Q , are as follows.

a. If we take 1 logf t t t in (1.1), we obtain

1

1 1, log ,

2 2 2

ni i

f i i R

i i

p qS P Q p q J P Q

q

, (1.2)

where ,RJ P Q is called the Relative J- divergence [12].

b. If we take 1

1 1 , 0,1s

sf t s s t t s R

in (1.1), we obtain


3

1

1

, 1 12

sn

i is i

i i

p qQ P s s q

q

2

1

1

01

11

2

2

2

1

1 1, , , 1

4 4

2log , lim , , 0

log , lim , , 12 2

1 1, , , 2

8 8

ni i

i i i

n

i

i ss

i i i

n

i i i i

ss

i i

ni i

i i

p qP Q Q P s

p q

qq F Q P Q P s

p q

p q p qG Q P Q P s

q

p qP Q Q P s

q

(1.3)

where , , 0,1s Q P s R is one parametric generalized divergence measure, which is known as

adjoint of the Unified Relative Jensen- Shannon and Arithmetic- Geometric divergence measure of type ‘

s ’ ,s P Q [32], s is called parameter and ,P Q is the Triangular discrimination [11], ,F Q P is

adjoint of the Relative Jensen- Shannon divergence ,F P Q [29], ,G Q P is adjoint of the Relative

Arithmetic- Geometric divergence ,G P Q [31], and 2 ,P Q is the Chi- Square divergence (Pearson

div. measure) [25].

Similarly, we can obtain many divergences by using linear convex functions. Since these divergences are

not worthful in practice, therefore we can skip them.

Now, there are two generalized means which are being used in this paper for calculations only. This are as

follows.

1 1

, 1, 01

log log, , 1 , 0,

1, 0

p p

p

b ap

p b a

b aL a b p a b a b

b a

p

. (1.4)

1

1,

, , 0

,

b b a

a

ba b

I a b a be a

b a b

. (1.5)

Means (1.4) and (1.5) are called p - Logarithmic power mean and Identric mean respectively.


4

2. New information divergence measure, properties and relations

In this section, we introduce new information divergence measure of class ,fS P Q . Properties and

relations of this new divergence with other divergences are also given. Let us start with definition of

convex function.

Convex function: A function f x is said to be convex over an interval ,a b if for every 1 2, ,x x a b

and 0 1 , we have 1 2 1 21 1f x x f x f x , and said to be strictly convex if

equality does not hold only if 0 or 1 .

The following properties (Theorems 2.1, 2.2, and 2.3) of new generalized f - divergence ,fS P Q and

their proofs can be seen in literature [18], so we omit the details.

Theorem 2.1 (Positivity) Let :f R R be a real, convex function and , n nP Q , then we have

, 1f

S P Q f . (2.1)

If f is normalized, i.e., 1 0f then , 0fS P Q , and if f is strictly convex, equality holds in (2.1) if

and only if 1, 2, 3,...i ip q i n , i.e., , 0fS P Q and , 0fS P Q if and only if P Q .

Theorem 2.2 (Linearity property) If 1f and 2f are two convex functions such that 1 2F af bf then

1 2

, , ,F f fS P Q a S P Q b S P Q , where a andb are constants and , n nP Q .

Theorem 2.3 (Relation with Csiszar’s f - divergence) Let :f R R be a convex and normalized

function, i.e., 0 0f t t and 1 0f respectively then for , n nP Q , we have

, , ,ff f C

S P Q C P Q E P Q , (2.2)

where ,fS P Q is given by (1.1), ,fC P Q is well known Csiszar’s f - divergence, given by

1

,n

if i

i i

pC P Q q f

q

, (2.3)

and

1

,f

ni

C i i

i i

pE P Q p q f

q

. (2.4)

Now, let :f R R be a mapping defined as

1 1

log2 2

t tf t

t

, (2.5)


5

1 1 1

log , 1 02 2

tf t f

t t

and

2

10

2 1f t t R

t t

. (2.6)

Since 0 0tf t and 1 0f , so f t is convex and normalized function respectively. Now put

f t in (1.1), we get

*

1

3 3, log ,

4 2

ni i i i

f

i i i

p q p qS P Q M P Q

p q

. (2.7)

We conclude the followings for new divergence * ,M P Q .

a. In view of theorem 2.1, we can say that * , 0M P Q and is convex in the pair of probability

distribution , n nP Q .

b. * , 0M P Q if and only if P Q or i ip q (attending its minimum value).

c. * * *, , ,M P Q M Q P M P Q is non- symmetric divergence measure.

Now, we derive two simple relations (Propositions 2.1 and 2.2) for * ,M P Q .

Proposition 2.1 Let , n nP Q , then we have the following new inter relation

* 1, , ,

4M P Q F Q P G Q P , (2.8)

where * ,M P Q is given by (2.7) and , , ,F Q P G Q P together are given by (1.3).

Proof: Since we know that AM GM , i.e., for , 0a b

2

a bab

. (2.9)

Now put 1a and1

bt

in (2.9) for 0t , we get

1/21

11 1 1 1 1 1

log log2 2 2 2

t tt

t t t t t

. (2.10)

Now multiply (2.10) by1

2

t for 0t , we get


6

1 1 1 1log log

2 2 4

t t t

t t

. (2.11)

Now put , 1,2,3,...,2

i i

i

p qt i n

q

in (2.11), multiply by iq and then sum over all 1,2,3,...,i n , we get

1 1

3 3 3 2log log

4 2 8

n n

i i i i i i i

i ii i i i

p q p q p q q

p q p q

, i.e.,

*

1

2 21, log

4 2

n

i i i i

i i i

p q q qM P Q

p q

, i.e.,

*

1 1

21, log log

4 2 2

n n

i i i i i

i

i ii i i

q p q p qM P Q q

p q q

, i.e.,

* 1, , ,

4M P Q F Q P G Q P . Hence prove the inequality (2.8).

Proposition 2.2 Let , n nP Q , then we have the following new inter relation.

* 21, , ,

2RM P Q J Q P Q P , (2.12)

where * , , ,RM P Q J Q P and 2 ,Q P are given by (2.7), (1.2), and (1.3) respectively.

Proof: Since we know the following by (2.2).

1 1

, ,2f

n ni i i

f C i i i

i ii i

p q pS P Q E P Q q f p q f

q q

. (2.13)

Now put f t and f t in (2.13), we get

1 1

3 3 1log log

4 2 2 2

n ni i i i i i i

i i

i ii i i i

p q p q p q qp q

p q p p

, i.e.,

*

1 1

1, log

2 2

n ni i ii i

i i

i ii i

q p qp qM P Q p q

p p

, i.e,

*

1

1, ,

2

ni i i

R

i i

q p qM P Q J Q P

p

, i.e.,


7

2

*

1

1, ,

2

ni i i i i

R

i i i

q p p q pM P Q J Q P

p p

, i.e.,

* 21, , ,

2RM P Q J Q P Q P ,

1 1

1n n

i i

i i

p q

.

Hence prove the inequality (2.12).

Remark: By taking together (2.8) and (2.12), we get

* 21 1, , , , ,

4 2RF Q P G Q P M P Q J Q P Q P .

3. New information inequalities

In this section, we introduce two new information inequalities (Theorems 3.1 and 3.2) on ,fS P Q ; one

of them is in terms of one parametric divergence ,s Q P . Such inequalities are for instance needed in

order to calculate the relative efficiency of two divergences.

Theorem 3.1 Let :f R R be a real, convex function on , R with 0 1 , .

If , nP Q and satisfying the assumption1

0 1,2,3,...,2 2

i i

i

p qi n

q

, then we have

the following inequalities

, ,f fS P Q B , (3.1)

where ,fS P Q is given by (1.1) and

1 1

,f

f fB

. (3.2)

Proof: Since f is convex on 0, , therefore we can write the following for , , 0,1R R

by the definition of convex function

1 1f f f . (3.3)

Now assumex

for ,x in (3.3), we get

x f x f

f x

. (3.4)


8

Now put2

i i

i

p qx

q

in (3.4), multiply by iq and then sum over all 1,2,...i n , we obtain the require

inequality (3.1).

Theorem 3.2 Let :f R R be a real, convex and twice differentiable function on , R with

0 1 , . If there exists the real constants ,m M with m M and

2 sm t f t M for all , , 0,1t s R .

If , nP Q and satisfying the assumption1

0 1,2,3,...,2 2

i i

i

p qi n

q

, then we have

, , , , , ,s ss f f sm B Q P B S P Q M B Q P , (3.5)

where ,fS P Q , ,s Q P are given by (1.1) and (1.3) respectively, and

1 1

,f

f fB

. (3.6)

1 1

,s

s sB

. (3.7)

Proof: Let us define a function :mF R R as

1

1 1s

m sF t f t m s s t f t m t

, (3.8)

where 1

1 1s

s t s s t

, 0,1s R . (3.9)

Since f t and s t are both twice differentiable, therefore mF t is twice differentiable as well, So

2 2 2 0s s s

mF t f t mt t t f t m .

Since 0 , 0,mF t t , therefore mF t is convex as well.

Now we write inequality (3.1) for the function mF t , we obtain

, ,m mF FS P Q B , i.e.,

1

1 1

1 12 2

sn n

i i i ii i

i ii i

p q p qq f m s s q

q q


9

1 1 1 1 11 11

s sf f

m s s

, i.e.,

, , , ,sf s fS P Q m Q P B mB , i.e.,

, , , ,s s f fm B Q P B S P Q .

Hence prove the first inequality of (3.5).

The second inequality of (3.5) obtains by a similar approach for the function

1

1 1s

MF t M s s t f t

. We omit the details.

4. Bounds of the new information divergence measure

In this section, we obtain bounds of the new information divergence measure * ,M P Q in terms of one

parametric generalized divergence measure ,s Q P for different values of s , by using new inequalities

(3.5) and means (1.4) and (1.5).

Proposition 4 Let ,s Q P and * ,M P Q be defined as in (1.3) and (2.7) respectively. For

, nP Q , we have

a. For 0,1s and 0s , we have

*1

, , , ,2 1 s s fs

B Q P B M P Q

1, ,

2 1 s ssB Q P

. (4.1)

b. For 1s , we have

*1

, , , ,2 1 s s fs

B Q P B M P Q

1, ,

2 1 s ssB Q P

, (4.2)

where ,fB and ,s

B are evaluated below by equations (4.4) and (4.5) respectively.

Proof: Put f t from (2.5) into (1.1), (3.6) and s t from (3.9) into (3.7), together with considering

mean (1.4), we get the followings respectively.


10

*

1

3 3, log ,

4 2

ni i i i

f

i i i

p q p qS P Q M P Q

p q

. (4.3)

1 1

,f

f fB

=

1 11 1 log 1 1 log

2 2

2

. (4.4)

1 1

,s

s sB

= 1 1 1 1 1

1

s s

s s

= 1 1

1

1

s s s s

s s

= 1

1 21 , 1 , 1 , 0,1s ss s sL s L s R

. (4.5)

Now let us consider the function

2 1

2 1

s

sg t t f t

t t

, where f t is given by (2.6), and

21

0, 01

0, 12 1s

ss t sg t

st t

. (4.6)

So g t is monotonically decreasing for 0s and monotonically increasing for 1s .

Therefore, we have

,

1, 0

2 1inf

1, 1

2 1

s

t

s

g s

m g t

g s

. (4.7)


11

,

1, 0

2 1sup

1, 1

2 1

s

t

s

g s

M g t

g s

. (4.8)

The inequalities (4.1) and (4.2) are obtained by using (4.3), (4.4), (4.5), (4.7), and (4.8) in (3.5).

Now we consider some special cases of proposition 4 at 1, 0, 1,s s s and at 2s for getting

bounds of the new divergence measure * ,M P Q , in terms of other divergences.

Result 4.1 Let ,P Q and * ,M P Q be defined as in (1.3) and (2.7) respectively. For , nP Q , we

have

*

3 2

12 , , 1 , , ,

4 1 2fL L P Q B M P Q

3 2

12 , , 1 ,

4 1 2L L P Q

. (4.9)

Proof: We evaluate ,s Q P and ,s

B at 1s , i.e.,

1

2 2

1

1 1 1 1

21 1 1, 1

2 2 2 2

n n n ni i i i i i i i i i

i i

i i i ii i i i i

p q q q p q p q p qQ P q q

q p q p q

=

1 1 1 1 1

1 1 12 1 2 2

2 2 2 2

n n n n ni i i i i i i i i i i

i i i i ii i i i i i i i

q p q p q p q p q p q

p q p q p q p q

=

2 2

1 1

41 1 1,

4 4 4

n ni i i i i i

i ii i i i

p q p q p qP Q

p q p q

. (4.10)

1 3 2

1, 2 , , 1

2B L L

. (4.11)

After putting (4.10) and (4.11) together with (4.4) in (4.2) at 1s , we get the result (4.9) in terms of

Triangular discrimination.

Result 4.2 Let ,F Q P and * ,M P Q be defined as in (1.3) and (2.7) respectively. For , nP Q , we

have

*

1

1 1 1log , , 1 , , ,

2 1fI L F Q P B M P Q


12

1

1 1 1log , , 1 ,

2 1I L F Q P

. (4.12)


B at 0s , i.e.,

1

00 0

1

, lim , lim 1 12

sn

i is i

s si i

p qQ P Q P s s q

q

1

2log ,

ni

i

i i i

qq F Q P

p q

. (4.13)

0 0

1 1 1 1 0, lim

1 0

s s

sB

s s

.

After applying D’ Hospital Rule, we obtain

0 0

1 log 1 log log log log log, lim

2 1

s s

sB

s

= 1

1 1log , , 1I L

. (4.14)


Relative Jensen- Shannon divergence.

Result 4.3 Let ,G Q P and * ,M P Q be defined as in (1.3) and (2.7) respectively. For , nP Q , we

have

*

1

1log , , 1 , , ,

2 1fI L G Q P B M P Q

1

1log , , 1 ,

2 1I L G Q P

. (4.15)


B at 1s , i.e.,

1

11 1

1

, lim , lim 1 12

sn

i is i

s si i

p qQ P Q P s s q

q


13

1

log ,2 2

ni i i i

i i

p q p qG Q P

q

. (4.16)

1 1

1 1 1 1 0, lim

1 0

s s

sB

s s

.

After applying D’ Hospital Rule, we obtain

1 1

1 log 1 log log loglog log, lim

2 1

s s

sB

s

= 1log , , 1I L . (4.17)


Relative Arithmetic- Geometric divergence.

Result 4.4 Let 2 ,P Q and * ,M P Q be defined as in (1.3) and (2.7) respectively. For , nP Q ,

we have

2 *

12

1 12 , 1 , , ,

4 1 4fL P Q B M P Q

2

12

1 12 , 1 ,

4 1 4L P Q

. (4.18)


B at 2s , i.e.,

2 2

2

1 1 1

1 1, 1

2 2 2 4

n n ni ii i

i i

i i ii i

p qp qQ P q p

q q

2

2

1

1 1,

8 8

ni i

i i

p qP Q

q

. (4.19)

2 1

1, 2 , 1

2B L . (4.20)


Chi- Square divergence.


14

5. Bounds of the Relative J- divergence

In this section, we obtain bounds of the Relative J - divergence measure ,RJ P Q in terms of one

parametric generalized divergence measure ,s Q P for different values of s , by using new inequalities

(3.5) and means (1.4) and (1.5).

Proposition 5 Let ,s Q P and ,RJ P Q be defined as in (1.3) and (1.2) respectively. For , nP Q

, we have

a. For 0s and 0s , we have

1 1, , , ,

2s

s s

s f RB Q P B J P Q

1 , ,s

s s

sB Q P . (5.1)

b. For 1s and 1s , we have

1 1, , , ,

2s

s s

s f RB Q P B J P Q

1 , ,s

s s

sB Q P , (5.2)

where ,fB is evaluated below by equation (5.6) and ,s

B is given by (4.5).

Proof: Let : 0,f R be a function defined as

1 logf t t t . (5.3)

1

log , 1 0t

f t t ft

and

2

1tf t

t

. (5.4)

Since 0 0tf t and 1 0f , so f t is convex and normalized function respectively.

Put f t from (5.3) into (1.1) and (3.6), together with considering mean (1.4), we get the followings

respectively.

1

1 1, log ,

2 2 2

ni i

f i i R

i i

p qS P Q p q J P Q

q

. (5.5)


15

1 1 1 1 log 1 1 log,f

f fB

1

log log1 1 1 1 ,L

. (5.6)

Now let us consider the function 2 2 1

2

1s s s s

tg t t f t t t t

t

, where f t is given by

(5.4), and

10, 1

10, 0

s ss

g t st s ts

. (5.7)

So g t is monotonically decreasing for 1s and monotonically increasing for 0s .

Therefore, we have

1

1,

, 1inf

, 0

s s

s st

g sm g t

g s

. (5.8)

1

1,

, 1sup

, 0

s s

s st

g sM g t

g s

. (5.9)

The inequalities (5.1) and (5.2) are obtained by using (4.5), (5.5), (5.6), (5.8), and (5.9) in (3.5).

Now we consider some special cases of proposition 5 at 1, 0, 1,s s s and at 2s for getting

bounds of the Relative J - divergence measure ,RJ P Q , in terms of other divergences.

Result 5.1 Let ,P Q and ,RJ P Q be defined as in (1.3) and (1.2) respectively. For , nP Q , we

have

3 2

11 2 , , 1 , 2 , ,

2f RL L P Q B J P Q

3 2

11 2 , , 1 ,

2L L P Q

. (5.10)

Proof: After putting (4.10) and (4.11) together with (5.6) in (5.1) at 1s , we get the result (5.10) in

terms of Triangular discrimination. We omit the detail.

Result 5.2 Let ,F Q P and ,RJ P Q be defined as in (1.3) and (1.2) respectively. For , nP Q , we

have


16

1

1 12 1 log , , 1 , 2 , ,f RI L F Q P B J P Q

1

1 12 1 log , , 1 ,I L F Q P

. (5.11)


terms of Relative Jensen- Shannon divergence. We omit the detail.

Result 5.3 Let ,G Q P and ,RJ P Q be defined as in (1.3) and (1.2) respectively. For , nP Q , we

have

1

2 1log , , 1 , 2 , ,f RI L G Q P B J P Q

1

2 1log , , 1 ,I L G Q P

. (5.12)

Proof: After putting (4.16) and (4.17) together with (5.6) in (5.2) at 1s , we get the result (5.12) in terms

of Relative Arithmetic- Geometric divergence. We omit the detail.

Result 5.4 Let 2 ,P Q and ,RJ P Q be defined as in (1.3) and (1.2) respectively. For , nP Q , we

have

2

12

1 12 , 1 , 2 , ,

4f RL P Q B J P Q

2

12

1 12 , 1 ,

4L P Q

. (5.13)


terms of Chi- Square divergence. We omit the detail.

6. Numerical verification of results obtained

In this section, we give two examples for calculating the divergences *, , , , ,RJ P Q P Q M P Q and

verify the inequalities (Or bounds of * ,M P Q and ,RJ P Q in terms of ,P Q ) (4.9) and (5.10).

Example 6.1 Let P be the binomial probability distribution with parameters 10, 0.5n p and Q its

approximated Poisson probability distribution with parameter 5np for the random variable X ,

then we have


17

Table 1: 10, 0.5, 0.5n p q

ix 0 1 2 3 4 5 6 7 8 9 10

i ip x p .000976 .00976 .043 .117 .205 .246 .205 .117 .043 .00976 .000976

i iq x q .00673 .033 .084 .140 .175 .175 .146 .104 .065 .036 .018

2

i i

i

p q

q

.573 .648 .757 .918 1.086 1.203 1.202 1.063 .831 .636 .527

By using Table 1, we get the followings.

0.527 1.2032

i i

i

p q

q

. (6.1)

211

1

,i i

i i i

p qP Q

p q

0.0917. (6.2)

11

1

, log2

i iR i i

i i

p qJ P Q p q

q

0.0808. (6.3)

11*

1

3 3, log

4 2

i i i i

i i i

p q p qM P Q

p q

0.0076525. (6.4)

Put the approximated numerical values from (6.1) to (6.4) in (4.9) and (5.10), we get

3 *9.110 10 0.017059 0.007652 , 0.014414M P Q

and

1

0.04248 0.169675 0.0404 , 0.139912

RJ P Q

respectively and hence verified the inequalities (4.9) and (5.10) for 0.5p .

Example 6.2 Let P be the binomial probability distribution with parameters 10, 0.7n p and Q its

approximated Poisson probability distribution with parameter 7np for the random variable X ,

then we have


18

Table 2: 10, 0.7, 0.3n p q

ix 0 1 2 3 4 5 6 7 8 9 10

i ip x p .0000059 .000137 .00144 .009 .036 .102 .20 .266 .233 .121 .0282

i iq x q .000911 .00638 .022 .052 .091 .177 .199 .149 .130 .101 .0709

2

i i

i

p q

q

.503 .510 .532 .586 .697 .788 1.002 1.392 1.396 1.099 .698

By using Table 2, we get the followings.

0.503 1.3962

i i

i

p q

q

. (6.5)

211

1

,i i

i i i

p qP Q

p q

0.1812. (6.6)

11

1

, log2

i iR i i

i i

p qJ P Q p q

q

0.1686. (6.7)

11*

1

3 3, log

4 2

i i i i

i i i

p q p qM P Q

p q

0.0115412. (6.8)

Put the approximated numerical values from (6.5) to (6.8) in (4.9) and (5.10), we get

*0.01586 0.031810 0.0115412 , 0.02762M P Q

and

1

0.03991 0.22497 0.0843 , 0.1765882

RJ P Q

respectively and hence verified the inequalities (4.9) and (5.10) for 0.7p .

Similarly, we can verify the other inequalities (4.12), (4.15), (4.18), (5.11), (5.12), and (5.13).


19

7. Asymptotic approximation on new generalized f- divergence

In this section, we introduce asymptotic approximation of ,fS P Q in terms of well known divergence

2 ,P Q and get the approximate relations of * ,M P Q and ,RJ P Q with 2 ,P Q , respectively.

Theorem 7.1 If : 0,f R is twice differentiable, convex, and normalized function, i.e., 0f t

and 1 0f respectively, then we have

21

, ,8

f

fS P Q P Q

. (7.1)

Equivalently

2

, 1

, 8

fS P Q f

P Q

when P Q ,

where , 0 , i.e., , are very small and 2, , ,fS P Q P Q are given by (1.1) and (1.3)

respectively.

Proof: We know by Taylor’s series expansion of function f t at 1t , that

2

211 1 1 1 1

2

tf t f t f f t g t

, (7.2)

where

2

1 11 1 ...

3 4

t tg t f f

and we can see that 0g t as 1t , 1 0f

because f t is normalized, therefore from (7.2) we get

2

11 1 1

2

tf t t f f

. (7.3)

Now Put 2

i i

i

p qt

q

in (7.3), multiply with iq and then sum over all 1,2,...,i n , we get the desire

result (7.1).

Remark: Particularly if we take 1 1

log2 2

t tf t

t

and 1 logt t in (7.1), we get

* 21, ,

32M P Q P Q , 21

, ,2

RJ P Q P Q respectively, where ,RJ P Q and * ,M P Q are

given by (1.2) and (2.7) respectively.


20

8. Conclusion and discussion

In this work, we introduced new information divergence measure, characterized, defined the properties

and derived relations with other divergences. Asymptotic approximation of this new divergence and

Relative J- divergence has been derived as well. Two new information inequalities on ,fS P Q in terms

of standard one parametric generalized divergence have been introduced and further bounds of the new

divergence and Relative J- divergence have been obtained in terms of the other well known divergences

in an interval , , 0 1 with as an application of new information inequalities.

These bounds have been verified numerically by taking two discrete distributions: Binomial and Poisson.

We found in articles [3, 16] that square root of some particular divergences is a metric space but not each

because of violation of triangle inequality, so we strongly believe that divergence measures can be

extended to other significant problems of functional analysis and its applications, such investigations are

actually in progress because this is also an area worth being investigated. Such types of divergences are

also very useful to find the mutual information [12, 13, 17], i.e., how much information is conveyed of

one random variable about other and utility of an event [4, 30], i.e., an event is how much useful compare

to other event.

We hope that this work will motivate the reader to consider the extensions of divergence measures in

information theory, other problems of functional analysis and fuzzy mathematics.

References

[1] Bajaj R. K, Hooda D.S., “Generalized measures of fuzzy directed divergence, total ambiguity and

information improvement”, Journal of Applied Mathematics, Statistics and Informatics, vol. 6

(2010), no. 2, pp: 31- 44.

[2] Bassat M.B., “f- Entropies, probability of error and feature selection”, Inform. Control, vol. 39

(1978), pp: 227-242.

[3] Bhatia P.K, Singh S., “On a new Csiszar’s f- divergence measure”, Cybernetics and Information

Technologies, vol. 13 (2013), no. 2, pp: 43- 56.

[4] Bhullar J.S., Vinocha O.P., Gupta M., “Generalized measure for two utility distributions”,

Proceedings of the World Congress on Engineering, vol. 3 (2010), June 30- July 2.

[5] Bregman L.M., “The relaxation method to find the common point of convex sets and its

applications to the solution of problems in convex programming,” USSR Comput. Math. Phys.,

vol. 7 (1967), pp: 200 –217.

[6] Burbea J., Rao C.R, “On the convexity of some divergence measures based on entropy functions”,

IEEE Trans. on Inform. Theory, IT-28 (1982), pp: 489-495.

[7] Chen H.C., “Statistical pattern recognition”, Hoyderc Book Co., Rocelle Park, New York,

(1973).

[8] Chow C.K., Lin C.N., “Approximating discrete probability distributions with dependence trees”,

IEEE Trans. Inform. Theory, vol. 14 (1968), no 3, pp: 462-467.

[9] Csiszar I., “Information measures: A Critical survey”, in Trans. – In: Seventh Prague Conf. on

Information Theory, Academia, Prague, (1974), pp: 73-86.

[10] Csiszar I., “Information type measures of differences of probability distribution and indirect

observations”, Studia Math. Hungarica, vol. 2 (1967), pp: 299- 318.

[11] Dacunha- Castelle D., “Ecole d’Ete de Probabilites de”, Saint-Flour VII-1977, Berlin,

Heidelberg, New York: Springer, (1978).


21

[12] Dragomir S.S., Gluscevic V., Pearce C.E.M., “Approximation for the Csiszar’s f- divergence via

midpoint inequalities”, in Inequality Theory and Applications - Y.J. Cho, J.K. Kim, and S.S.

Dragomir (Eds.), Nova Science Publishers, Inc., Huntington, New York, Vol. 1 (2001), pp: 139-

154.

[13] Dragomir S.S., Scholz M.L., Sunde J., “Some upper bounds for relative entropy and

applications”, Computers and Mathematics with Applications, vol. 39 (2000), pp: 91- 100.

[14] Gokhale D.V., Kullback S., “Information in contingency Tables”, New York, Marcel Dekker,

(1978).

[15] Hooda D.S., “On generalized measures of fuzzy entropy”, Mathematica Slovaca, vol. 54 (2004),

no. 3, pp: 315- 325.

[16] Jain K.C., Chhabra P., “Series of new information divergences, properties and corresponding

series of metric spaces”, International Journal of Innovative Research in Science, Engineering

and Technology, vol. 3, no.5 (2014), pp: 12124- 12132.

[17] Jain K.C., Chhabra P., “New series of information divergence measures and their properties”,

Communicated.

[18] Jain K.C., Saraswat R.N., “Some new information inequalities and its applications in

information theory”, International Journal of Mathematics Research, vol. 4, no.3 (2012), pp:

295- 307.

[19] Jha P., Mishra V. K., “Some new trigonometric, hyperbolic and exponential measures of fuzzy

entropy and fuzzy directed divergence”, International Journal of Scientific and Engineering

Research, vol. 3 (2012), no. 4, pp: 1- 5.

[20] Jones L., Byrne C., “General entropy criteria for inverse problems with applications to data

compression, pattern classification and cluster analysis”, IEEE Trans. Inform. Theory, vol. 36,

(1990), pp: 23- 30.

[21] Kadota T.T., Shepp L.A., “On the best finite set of linear observables for discriminating two

Gaussian signals”, IEEE Trans. Inform. Theory, vol. 13, (1967), pp: 288-294.

[22] Kailath T, “The divergence and Bhattacharyya distance measures in signal selection”, IEEE

Trans. Comm. Technology, vol. COM-15, (1967), pp: 52- 60.

[23] Kazakos D., Cotsidas T, “A decision theory approach to the approximation of discrete

probability densities”, IEEE Trans. Perform. Anal. Machine Intell, vol. 1, 1980, pp: 61- 67.

[24] Nielsen F., Boltz S., “The Burbea-Rao and Bhattacharyya centroids”, Apr. (2010), Arxiv.

[25] Pearson K., “On the Criterion that a given system of deviations from the probable in the case of

correlated system of variables is such that it can be reasonable supposed to have arisen from

random sampling”, Phil. Mag., 50 (1900), pp: 157-172.

[26] Pielou E.C, “Ecological diversity”, New York, Wiley, (1975).

[27] Renyi A., “On measures of entropy and information”, Proc. 4th Berkeley Symposium on Math.

Statist. and Prob., 1 (1961), pp: 547-561.

[28] Santos-Rodriguez R., Garcia-Garcia D., Cid-Sueiro J., “Cost-sensitive classification based on

Bregman divergences for medical diagnosis”, In M.A. Wani, editor, Proceedings of the 8th

International Conference on Machine Learning and Applications (ICMLA'09), Miami Beach,

Fl., USA, December 13-15, (2009), pp: 551- 556, 2009.

[29] Sibson R., “Information radius”, Z. Wahrs. Undverw. Geb., (14) (1969), pp: 149-160.

[30] Taneja H.C., Tuteja R.K., “Characterization of a quantitative- qualitative measure of

inaccuracy”, Kybernetika, vol. 22 (1986), no. 5, pp: 393- 402.

[31] Taneja I.J., “New developments in generalized information measures”, Chapter in: Advances in

Imaging and Electron Physics, Ed. P.W. Hawkes, 91(1995), pp: 37-135.

[32] Taneja I.J., Kumar P., “Generalized non-symmetric divergence measures and inequalities”, The

Natural Science and Engineering Research Council’s Discovery grant to Pranesh Kumar (2000).

[33] Taskar B., Lacoste-Julien S., Jordan M.I., “Structured prediction, dual extra gradient and

Bregman projections”, Journal of Machine Learning Research, 7 (2006), pp: 1627- 1653.

[34] Theil H., “Statistical decomposition analysis”, Amsterdam, North-Holland, (1972).


22

[35] Theil H., “Economics and information theory”, Amsterdam, North-Holland, (1967).

[36] Vemuri B., Liu M., Amari S., Nielsen F., “Total Bregman divergence and its applications to

DTI analysis”, IEEE Transactions on Medical Imaging, (2010).

Documents

New Information Inequalities in Terms of One Parametric