Concentration places, concentration evolutions, and online information retrieval techniques for calculating them

InJormorion Processrng & Manqwnenr Vol. 24. No. 2, pp. 109-121, 1988 0306.4573/H 13.00 + .OO Printed in Great Britain. Copyright 0 1988 Pergamon Press plc

CONCENTRATION PLACES, CONCENTRATION EVOLUTIONS, AND ONLINE INFORMATION

RETRIEVAL TECHNIQUES FOR CALCULATING THEM

L. EGGHE LUC, Universitaire Campus, B-3610 Diepenbeek, Belgium* UIA, Universiteitsplein 1,

B-2610 Wilrijk, Belgium

(Received 20 November 1986; accepted in final form 22 July 1987)

Abstract-Suppose we have N unordered classes to which objects can belong (i.e., articles dealing primarily with a certain broad topic; we have then N topics). We define the concentration place relative to a certain permutation 4 of the set ( 1,. ,N]. We show that the concept is good in different ways. We then give a theory of concentration evolution resulting in a nontrivial problem in operations research. We calculate levels of evolutions as well as directions of evolutions. Once they are calculated, these data give valuable information on how interests in certain topics can change along the years. Com- parisons between the evolution of a journal and the evolution of the whole subject can be made as well as comparisons between the evolution of two journals. Also, the velocity of evolution can be measured with this theory.

An explicit examination is done for the Transactions of the American Mathemat- ical Society, in comparison with mathematics as a whole.

Keywords; Concentration place, concentration evolution.

1. INTRODUCTION

In [l], Pratt defines a measure q (in Pratt’s notation q,) as follows. Suppose we have N unordered classes (or boxes). Objects can be assigned to these classes in various ways. Take, for instance, the example of a subject classification: a big subject is divided into N subsections (classes); each article (object) is dealing primarily with exactly one of these subsections. In practice, these assignments are not difficult to establish: a lot of online sys- tems have the facility to determine the major topic an article deals with. We will work out a concrete example at the end of this article, which is a model for lots of other applications.

Once we know how many objects belong to each class, we can determine the relative frequency a, of each class. The measure q is then defined as

N

q = C ia,. i=l

q can be interpreted as the “expected class” (being an average) and hence can be used as a measure of concentration place. In [ 11, q is transformed into the so-called “Pratt measure” C, which is only a normalization of q in order to obtain a measure of concentration with values between 0 and 1. The measure C is, however, not used in here. In the above definition of q, we put the N classes in a fixed but arbitrary order (1,2, . . . , N). Of course, different orders give different values of q, and since the classes are unordered, they are as much important as the initial arbitrary order. Therefore, we define, if we denote by

7rN = (4114 is a permutation of 1,2,. . .,Nj,

for every 4 E 7~~.

*Permanent address

IPH 24:2-A 109

(2)

110 L. EGGHE

Denote q = qQ for 4 = I(,, the identity permutation. If this measure is to function as an indication of the place of concentration, we want to have the property that, if we take 4 = Id and if we take the “mirror image” of the distribution a,, a,, ,a&, that is, a N,aN_I,. ,a, (or a,’ = a+,+!, i = 1,. , N) we would like to find N - q as our new value of q (since we have in total N classes). This is not so, due to the fact that (see Fig. 1) the ith class has abscissa i at the end of this class, and not in the middle.

This problem can be solved by putting, for every o E T,~,

or:

(3)

since c a, = 1. We call P, the concentration place of the distribution, with respect to ,=I

C$ E i,: We denote PQ = P for 4 the identity permutation. That we can really detect the intuitive concentration place of a distribution, by choosing the right 4 E w,$, is seen in the next example. Suppose we have only two classes of the N classes occupied as follows: a, =a1 = i,a,=O, i=2 ,___, N- l.Then,

p=;+N_I=N 2 2 2’

But, if N 2 3, P indicates a place where we encounter only empty classes. Take now C#J E f,$ to be the following permutation: $(2) = N, 6(N) = 2, 4(i) = i, i = 1,3,4,. ,N - 1. Then we have

exactly between the two occupied classes. (See Fig. 1: the abscissa 1 is exactly between the first and the second class.)

This observation, although elementary, will become the main tool in our theory of concentration evolution: we will establish a method of finding an “optimal” $ E 7r,V such that, in this order of the classes, the evolution over k fixed time periods is best shown (is “maximized”). From this rl/, also measures of evolution will be derived.

Remark. This article is only dealing with the problem of measuring evolution from one situation to another and the concentration places in relation with this. If only one situation is the subject of study, one might as well take $ = I,, in eqn (3).

Let us, for the remainder of this section, summarize some properties of P$ (q5 E 7rTN

arbitrary but fixed) that show that the definition is good.

1. As we mentioned already, we have the following property: suppose 4 = Id and suppose we have a distribution (a, ,az, . ,av), yielding the concentration place P’. If we “mirror” this distribution to become (a,.,~,~_, , ,a, ) (i.e., u,~ ,+ , , i = 1, , N), then the concentration place P2 of this distribution is N - PI.

Concentration places and evolutions

Proof.

P2 = t ia+;+, - i i=l

Putj=N-i+l;hencei=N-j+l.Then

P2= i (N-j+ *Jai-i j=l

= N- Ejaj+ i j=l

=N-Pi.

The same property is true, for every 6 E K~ with the additional property $(N - i + 1) = N - q5(i) + 1.

2. Suppose ai = 1 and a, = 0, for every j # i. Then P+ = 9(i) - i. 3. Let 4 = Id. Suppose the first distribution (a/)i=l,, + has the values:

1-o a,! = - N- 1

for every j + i,, with a;, > aj(tlj # i,). Suppose the second distribution (a?);=,, ,N is the same as above but with i, replaced

by i2 with i2 > i,. Then P2 > P’.

Proof.

p2 = fJ ia,’ - i i=l

p2=ii eN 1=l i J +i2cx- k.

I#& (4)

Since, by definition [i2 - i,][cx - (1 - a)/(N - 1)] > 0, we have also i2cx > i2[(l - a)/(N - l)] + cri, - i, [(1 - cw)/(N - l)]. Hence (4) becomes:

P2> g i(c) +i,(s) +i2(fi) +ai, -i,(s) - i

ifl,,lZ

= ,g i(s) + ai, - i

ifi,

= P’.

4. Is 4 E ?T~ arbitrary, and if ai = l/N for every i = 1,. . . ,N, then P+ = N/2.

112

Proof.

L. EGCHE

But 9 E 7rN. Hence,

so:

,$ G(j) = t j = N(Nz+ ‘) . !=I

2. CONCENTRATION EVOLUTION AND CORRESPONDING

CONCENTRATION PLACES

Suppose again that we have the situation of N classes, each occupying a fraction a,(i = 1,. . . ,N). In the previous section we have seen that, on applying an appropriate

permutation 4 E aN, we can find a good notion of concentration place, namely

.v

Pd = C 4(i)aj - i. I=1

(3)

Here we suppose that-although the N classes are unordered-we fix a certain order on these N classes on which 4 is applied. Our theory, which we will develop now, will be inde- pendent on the initial (arbitrary) order that we use.

Usually, the fractions a, are dependent in time. Hence, with time, concentration changes, and hence also the concentration place according to a fixed 4 E r,,,. It is now our purpose to find an “optimal” $ E aN such that evolution from one time period to

another (or between several time indicators) is best seen. This is worked out in the next two sections.

2.1 The case of two points in time This represents the simplest case. We have here two points in time, t, and tz, and we

want to see how the distribution of the occupancy of the N classes changes from f, to t2 (i.e., how there is a change in publication habit of a journal over N subject classes between times t, and tz). Let us denote by (a,] ),=,, ,,1’ and (a,‘),=,, ,N the fractions of occupancy on the respective times t, and t2.

The evolution from t, to tz is best seen if we use the permutation 4 E rh: with the property:

Pi - PJ = max (Pi -Pi). GbEir,

(9

In this way, classes are moved to the left if Ph - Pj is as high as possible and they are moved to the right if Pz - Pi is as high as possible. Hence, the first classes become less popular in time (although they can still be the biggest-this has nothing to do with it, since in (5) we use substractions), whereas the last classes become more popular in time. We hence say that the classes, when put in the order

$(1),$(2),. . .,$(N)

Concentration places and evolutions 113

give the natural evolution form time tr to time f 2. For this algorithm, no computer is needed; the determination of $ as “optimal” permutation is seen at once:

P$ - Pi = max f $(i)(a,2 - a/) @Ea,V ,=,

So the lowest a: - u; gives $(i) = 1. The second lowest a,’ - u) gives 4(i) = 2, and so on; the biggest af - a,] gives $(i) = N. For a concrete example, see the next section.

2.2 The general case of k points in time We deal now with the case of k times t,, tS, . . . , tk between which we want to deter-

mine the evolution of the occupancy of the N classes. We suppose that the fractions of occupancy at each time tj(j = 1, . . . , k) are known, being (aj);=r,. ,,,, j = 1, . . . , k.

Now we encounter several additional problems, in comparison with the previous case. First of all, determining

k-l

max C (Pi+’ - Pi) = max (P,” - Pd) dE*N j= I 4Gnf-d

(6)

is not good since the intermediate points in time are left out of consideration. With (6) we just apply the previous case for two times t, and tk.

One way to overcome this difficulty is to use the power 2 in (6) as follows: choose $ E TN such that

k-l

However, this is not the good algorithm. This can be seen if we apply this algorithm to the simplest case of k = 3, and

a: = 1,

ai= 1,

a: = 1 9

Algorithm (7) then yields

a: = 1,

a;= 1,

az’ = 1 9

.! =o I 9

a? = 0,

a’ = 0,

a; = 0,

a,2 = 0,

a,3 = 0,

i=2,...,N

i = 1,3,. . . ,N

i = 1,2,4,. . . ,N.

i=2,...,N

i = 1,. . ,N - 1

i = 1,3,. . . ,N

(seeFig. 2). (Hence$(l) = 1, G(2) =N,$(3) =2, G(i) =i(i=4,5 ,..., N- l), $(N) = 3.) There are a few other permutations that give the same situation; the solution is not always unique, especially with simple examples as above; in the real example in the next section we will see that most likely, $ is unique.

The situation in Fig. 2b is not what we want. What we have in mind to see evolution is Fig. 2c, which shows in a perfect way the evolution from tl over t2 to t3.

An obvious requirement is that

114

1

L. EFGHE

Given situation

1

I

Situation when (2: 1 L I I I I I 1 I

wr&ing (7) 0 I 2 3 N-l N

Wanfed situation

Fig. 2b.

.I.

0 I 2 3 N - 1 N

1

.L.

0 1 2 3 hi - I N

13: ’ I I I 1 I I I I I

0 1 2 3 N-l N

Fig. 2~.

So, we formulate our algorithm as follows. Let

?r~=r~E~NIIP~rp~~..,~plc”).

Then we look for 4 E a,$ such that

k-l k-l

A2 = c (Pi” - Pi)” = max c (f’,$+’ - P,j)’ j=l 4EE& j=1

=

(8)

(9)


I

Remarks:

1.

2.

3.

4.

5.

In case k = 2, definition (9) gives the same 4 as in definition (5) of the previous section; indeed, (5) implies that Pk, 5 Pi (if not, mirror c$! ) and if +G is such that (P$ - P$)’ = max (Pz - Pd)’ then we obviously have also (5) for the same I/.

Instead of algorithm (9), we cannot use the algorithm

max C C 4(i)(aJ+’ - a:)2 6ET h ,=I ,=I

(10)

In case of Fig. 2a, this would yield the unwanted situation of Fig. 2d (or an equiv- alent one, giving the same maximum in (lo), namely 4N - 4).

In Fig. 2d we even have Pi 5 P,j 5 P$ so that also the following algorithm is not good:

k-l 8%

max C C qS(i)(a,‘+’ - a,J)* dEnL ,=I ,=,

(11)

A* is a measure of how big the evolution is whereas Pi, P$, . . . , P$ indicate “how” the evolution was established: one simply checks the relative distances Pi+’ - P$(j = 1,. . ,k - 1).

Algorithm (9) is programmed for a personal computer (on Apple 11~). This program can be received on request from the author. This program has been used in the elaboration of the example in section 3. The algorithm expressed in eqns (8) and (9) is, in fact, a quadratic integer (1-O) programming problem. This can be seen as follows, as was communicated to me by R. Rousseau: Put X,j = 1 if the original ith group is changed on to thejth place and Xlj = 0 if this is not so. We then have the following algorithm:

vi= l,... ,N: iX,j= 1

vj= I,... ,N: ;x;, = 1 i=l

N N

Vs = 1,. ,k - 1: c j,qj-ai(‘) I c jxiJ.ai(‘+‘) l.J=l i,J=l

116 L. EGCHE

and the problem is to maximize

j(~,‘~+” - Q,(s))x,,

So, our algorithm is a quadratic integer (1-O) programming problem.

2.3 Artificial example of the case k = 3 We suppose we have the following situation (Fig. 3a): N = 7, k = 3 and

a: = ai = ai = ai = 0: = 0, 1

a< =a’= ~ 6 2

af = aa = a;? = 0; = 0, 7 1 0: = af = u,- = ~

3

a: =a3 - 3 -a~=a~=a~=(); ai’ =a2 = ! -3

If we apply algorithm (9) then we find the situation shown in Fig. 3b: G(5) = 1, $(6) = 2, $(2) = 3, $(3) = 4, 11/(7) = 5, $(I) = 6, $(4) = 7. This is exactly the required situation.

This shows that we have found the exact algorithm to determine concentration evolutions. In the next section, this algorithm (9) will be applied to a real-life example.

I i’2 I /‘2 I

0 1 2 3 4 5 6 7

0 1 2 3 3 5 6 7

l/2 l/2

1 I I I I

0 I 2 3 4 5 6 I

Fig. 3a.

l/2 l/2

0 1 2 3 4 5 6 7

l/3 , li3 , I/3

L 1 1 I , 8 0 1 2 3 4 5 6 7

I/2 l/2

I 1 I 1 I 0 1 2 3 4 5 6 7

Fig. 3b


3. APPLICATION

3.1 The data The American Mathematical Society (AMS) uses, as is well known, the AMS classi-

fication codes. As a lot of other codes in other disciplines, they are treelike structured. The broadest groups are obtained by truncation after the first digit. In DIALOG, the command

for such a broad group is DC = 0 ?, DC = l?, and so on, until DC = 9?. Although mathematical articles contain several AMS codes, they select only one pri-

mary classification code. In this way, articles are uniquely assigned to one of the 10 major classes, described above. To retrieve them online, DIALOG has the suffix /MAJ, to restrict to articles dealing primarily with the topic in question.

We want now to examine the possible evolution of the fraction of the articles in each broad class, and we do this over the years 1975, 1980, and 1985. This evolution will be measured for mathematics as a whole, and for one journal in particular, the Transactions of the American Mathematical Society.

So we create three groups of commands online: First group. S DC = 0?/MAJ, S DC = l?/MAJ, . . . ,and so on until S DC =

9?/MAJ. Second group. S PY = 1975, S PY = 1980, S PY = 1985. Third group. After expanding E TRANS it became clear that, when using the com-

mand S JN = TRANS. AMER. MATH. SOC. we obtain exactly 4000 of the 4012 articles in this journal. So we used the above command.

All mutual intersections (i.e., AND relations) between the commands of group 1 and of group 2 give the total number of articles (in mathematics as a whole) that deal primarily with the respective subject classes of group 1 and this for the respective years, mentioned

in group 2. These data are collected in Table 1. One further intersection with the command in group 3 gives the same data but now for the Transactions of the American Mathemat- ical Society only. These data are collected in Table 2.

Since the last three classes in Table 2 stand for maximally 2% of all articles, we add these with class 7. Tables 1 and 2 then become Tables 1’ and 2’, giving relative frequen- cies now:

3.2 Evolution 1975-1985 Here we use only the data for 1975 and 1985, applying Section 2.1 (k = 2). In this

case we can determine the optimal $ E a, immediately, without needing computer time. 3.2.1 All articles. From Table 1’ we determine the difference table:

Class 1 2 3 4 5 6 7

n 1985-n 1975 -0,008 -0,018 -0,009 -0,013 -0,029 0,001 0,075

So A is maximal for II/ E r-i as follows: $(5) = 1, $(2) = 2, $(4) = 3, $(3) = 4, $(l) = 5, $(6) = 6, G(7) = 7, yielding:

P; - Pi = -0,029 - 2.0,018 - 3.0,013 - 4.0,009 - 5.0,008 + 6.0,OOl + 7.0,075

Here

= 0,351

A2 = (Pi - Pi)* = 0,123

118 L. EGGHE

Table 1.

Class Section n 1975 n 1980 fll985

1 DC = O?/MAJ 3403 4013 3814 2 DC = l?/MAJ 3078 3365 3026 3 DC = Z?/MAJ 1858 2074 1902 4 DC = 3?/MAJ 4132 4722 4464 5 DC = 4?/MAJ 3999 4369 3669 6 DC = S?/MAJ 3487 4048 4233 7 DC = 6?/MAJ 543 1 699 I 6814 8 DC = 7?/MAJ 1429 3248 2554 9 DC = B?/MAJ 2268 4255 4504

10 DC : 9?/MAJ 3719 4888 4606

32804 41973 39586

Table 2

Class Section rAMS TAMS TAiMS n 1975 nl980 n 1985

I DC = 0?/MAJ 23 2 DC =. l?/MA.I 66 3 DC :- Z’?,‘MA.I 53 4 DC = 3?/MAJ 3x 5 DC = I?/MA.I 53 6 DC = 5‘?/MA.I X6 7 DC = 6?/MAJ 9 8 DC = 7?iMAJ 0 9 DC = 8?/MA.I 2

10 DC : 9?/hlAJ O

330 I

17 24 25 37 71 60

J 0 3 I

197

34 3s 37 56 57 XI I2 3

317

Table 1’

Class n 1975 n1980 fll985

1 0,104 0,096 0,096

2 0,094 0,080 0,076

3 0,057 0,049 0,048 4 0,126 0,113 0,113

5 0,122 0,104 0,093

6 0,106 0,096 0,107

7 0,392 0,461 0,467

Table 2’

Class TAMS TAMS TAMS n 1975 n1980 n 1985

I 0.070 0,086 0,107 2 0,200 0,122 0,110

3 0,161 0,127 0,117 4 0,115 0,162 0,177 5 0,161 0,157 0,180 6 0,261 0,305 0,256 7 0,033 0,041 0,054


and the concentration places with respect to I+C are

Pi = 4,316

P; = 4,667

The new order of the classes (illustrating best the evolution) is:

5-2-4-3-l-6-7

3.2.2 TAMS From Table 2’ we have now the following difference table:

Class 1 2 3 4 5 6 7

l-ll985-nl975 0,037 -0,090 -0,044 0,062 0,019 -0,005 0,021

Here, A is maximal for II, E 7r7 as follows: g(2) = 1, g(3) = 2, $(6) = 3, G(5) = 4, G(7) = 5, $(l) = 6, $(4) = 7. In this case:

P.$ - Pi = 0,644

A2 = 0,415

P$ = 2,839

Pi = 3,483.

The best order of the classes for illustrating the evolution in TAMS over 1975-1980 is now

2-3-6-5-7-l-4.

We can conclude here that, although TAMS is changing faster than mathematics as a whole (cf. compare the A values), TAMS is not evolving in the direction of the whole mathematics, which evolves in the direction of class 7 (fringe topics in mathematics).

Remark. In the above example, one cannot compare the values Pi for the cases: “all articles” (3.2.1) with the one for TAMS (3.2.2), since we are dealing with different permutations $ in those two cases. Only the values of Pi and P$ within each case are of impor- tance here, and along with it the value of A. Furthermore, the values of A for “all articles” and for “TAMS” can be and are compared, as explained above.

If one really wants to compare concentration places between the two cases (3.2.1 and 3.2.2) above, one must choose one $; for instance $ = 1,. If we do this in the above example, we find for “all articles”

P1 = 4,457 and P2 = 4,720

and for “TAMS”

P' = 3,515 and P2 = 3,700.

Of course, by our method, all values Pz - P’ are smaller than the P$ - Pi above, but still one sees the same trend (in the sense that P2 - P’ and P$ - Pi have the same

120 L. EGGHE

sign in both cases). However, this is not always the case. Indeed, for every permutation 4, there is a permutation $ such that

Pj - P; = Pi - Pi

(just apply property 1 in Section 1). So half of all permutations show a positive Pi - Pi

and half of them a negative one. To which group $ = Id belongs is not clear and hence $ = Id is not the permutation to base conclusions on. This is precisely where our method comes in: maximization of Pi - Pi shows in the clearest way the evolution in time and hence detects the natural order in which the evolution took place.

We now come to the case where we study the evolution from 1975, over 1980, to 1985. Intuitively it is clear that, if the fractions in all the classes for 1980 are intermediate between those for 1975 and 1985, we should find the same evolution order. In this case, however, the study over the three years remains useful, for example, to measure the difference of speed of evolution in the two time periods (cf. remark 3 of Section 2.2). Of course, if there are fractions of classes in 1980 that are not intermediate between those for 1975 and 1985, we also expect a different evolution order. These intuitive ideas will be examined now.

3.3 Evolution 1975-1980-1985

For this we wrote a computer program. For both Tables 1’ and 2’, the computer calculates, for every C#J E 7r7:

Ph = C +(i)a,’ - f /=I

If Pd 5 Pi 5 P,‘, then the system also calculates

A; = (Pi - P;)’ + (Pi - P,‘)’

and in this case one receives the printout of Q(i) (i = 1,. . ,7), Pi, Pi, P: and A$. It is then easy to select max As = A’. The numbers $(i) (i = 1, . ,7), P$, P,$, P$ and A2 give

tiEa: then all the information we need (cf. also remark 3 in Section 2.2).

3.3.1 All Articles

We find here:

$(5) = 1, $(2) = 2, $(4) = 3, G(3) = 4, $(S) = 1, $(6) = 6, $(7) = 7,

exactly the same solution as in 3.2.1. This was expected since in Table 1’ all values for 1980 are intermediate to those for 1975 and 1985 (except for the value in class 6, but the difference is small and hence did not affect the final evolution order), A2 here is small, being 0,07798. So the ideal order is

5-2-4-3-1-6-7.

The values for the concentration places are

Pi = 3,816, Pi = 4,082, P,j = 4,167.


Hence we see that the evolution is slow (this could also be seen from the value of A’) and that the evolution was faster in the period 1975-1980 than in the period 1980-1985 (since Pi - Pi < Pi - Pi). It is surprising that our method is good for giving concentration places, concentration evolution, total evolution speed, and evolution speed in every period.

3.3.2 TAMS Here we find

G(2) = 1, $(3) = 2, G(5) = 3, $(7) = 4, $(l) = 5, ti(6) = 6, $(4) = 7,

hence the order

2-3-5-7-l-6-4.

In the case 2.3.2 we had

2-3-6-5-7-1-4.

So only class 6 moved from the third place to the second last place. Something like this could be expected if we look at Table 2’: all values for 1980 are intermediate between those for 1975 and 1985 except for classes 5 and 6. However, the irregularity in class 5 is very small, but for class 6 it is substantial: 0,261 + 0,305 + 0,256. This influenced the different position of class 6. As before, we can conclude that TAMS evolves differently than mathematics as a whole.

Furthermore, the values for the concentration places are

Pi = 2,858, P$ = 3,405, P,3 = 3,41 and A2 = 0,29923.

From this we conclude (as in 3.2.1) that TAMS evolves faster than mathematics in its whole (but in another direction: not in the informatics direction). Furthermore, there has been much more change in the period 1975-1980 than in the period 1980-1985 (since P$ - P.$ << P$ - PJ).

Remark. The value of A2 = 0,29923 was much bigger than the second largest A$; so the choice of $ was evident. This was not so for the whole mathematics. There was one Ai close to A2 = 0,07798, namely o E r,+ yielding the order:

4-2-5-3-l-6-7

and yielding A$ = 0,07781. We see that only classes 4 and 5 are interchanged and that the most important evolution (to class 7) is still detected.

Remark. The online method we used here to work out this example can stand for a model of how to proceed with such practical investigations. Most online databases have the feature to limit a publication to this topic (subject code) the work deals with primarily. Then mutual intersections of the commands in the three groups (mentioned above) can be done in those files. Application of the theory then gives the result for this area of research.

Acknowledgmen!s-I thank the referee and Dr. R. Rousseau for interesting comments on the content of this article.

REFERENCE

1. Pratt, A. D., A measure of class concentration. J. Amer. Sot. Inf. Sci., 28: 285-292; 1977.

Documents

Concentration places, concentration evolutions, and online information retrieval techniques for calculating them