A R K I V F 0 R M A T E M A T I K B a n d 3 n r 43

Communicated 5 June 1957 by HARALD CR/~.~I]~R and OTTo FROSTMAN

The significance probability of the Smirnov two-sample test

B y J . L. HOBOES, Jr.~

With 3 figures in the t ex t

1. Introduction

In 1939 N. V. Smirnov proposed the following rank-order test for the two-sample problem. Let x 1 . . . . , xm and Yl . . . . . Yn be samples of independent observations f rom populations with continuous distribution functions F and G, respectively. Form from the samples the empirical distribution functions Fm and Gn; tha t is, mFm(u) is the number of the observations xl, ..., x m which do not exceed u, with nGn(u) defined analogously. To test the hypothesis _P = G we use the statistic D = sup lF~(u) -

U

G~(u)[, large values of which are significant. We may without loss of generali ty assume m ~ n.

I t is clear tha t the signficance probabil i ty Pr {D >= d [ F = G}, which we shall denote throughout by P, , is independent of the common value of F = G; tha t is, the test (like all rank-order tests for the two-sample problem) is similar over the class of all continuous distributions. Further, the fact tha t sup [F~(u) - F ( u ) [ tends to 0

in probabil i ty as m - ~ c ~ implies tha t the test is consistent against all alternatives F ~= G. These properties of similarity and consistency, together with a certain mathe- matical elegance, give the test wide appeal to mathematical statisticians. A consider- able literature has developed, the proposer of the test has been awarded a Stalin prize (Kolmogorov and HinOin 1951), and the test has begun to appear in applied handbooks. The test is not very powerful against specific alternatives such as shift (van der Waerden 1953), but this could hardly be expected in view of its consistency.

Smirnov's test was suggested by analogy with the earlier test of Kolmogorov (1933) for the one-sample problem. In fact, Smirnov's test generalizes Kolmogorov's , for when n--> c~ we may replace G~ by G, and D becomes Kolmogorov's statistic for the hypothesis that F equals a completely specified G. Thus general results on the Smirnov test usually give (by the limit passage m-+oo) results on the Kolmogorov test. We shall not however a t t empt to discuss the significance problem for Kolmo- gorov's test, nor shall we take up the many variants of Smirnov's test which have been suggested.

Smirnov's test also appears in a one-sided version. We m a y use D e = sup [F~ (u) - U

Gn(u)] to test the hypothesis tha t F(u) ~_ G(u) for all u. This form of the test is in

'1 This p a p e r was w r i t t e n whi le the a u t h o r was a fel low of the J o h n S imon G u g g e n h e i m m e m o r i a l founda t ion , a n d a gues t of S t e c k h o l m s hOgskola. I t is a p leasure to record a p p r e c i a t i o n for t h e cour tes ies e x t e n d e d to m e b y t h e hOgskola a n d i t s rec tor , Professor H a r a l d CramOr.

34 469

J. L. HOD¢~.S, JR., The Smirnov two-sample test

fact often appropriate when we wish to know whether a new method of t rea tment (producing the population G) gives larger values than the standard t rea tment (popula- tion F) for some quantile. We shall denote by P1 the quant i ty P r {D + > d [ F = G}, which is the size of the one-sided test. One can also define a non-symmetrical two- sided test statistic, which includes D and D + as special eases. While no essential difficulties appear in doing this, we shall not discuss the general statistic as its usefulness does not appear to justify the considerable notational complication required.

In spite of the fact tha t the Smirnov test has been in use for nearly twen ty years, the computat ion of P1 and P2 has not been satisfactorily dealt with, nor does it even seem to be widely realized tha t a computational problem exists. We review in Section 2 the methods now available, and give in Section 3 the results of a brief numerical investigation showing the inadequacy of those methods. In Section 4 we present a technique which is useful when m - n is small, and which serves to illuminate the complexity of the problem. Among the remarks in the concluding Section 5 is an interpolation formula which appears to give considerably bet ter values than the currently s tandard technique.

2. Methods for obtaining values of P1 and P2

The user of Smirnov's test, faced with the problem of obtaining a value of P1 or Pg. corresponding to the observed value of D + or D, has available a var ie ty of methods, which we shall now review.

(a) Direct computation As the distribution F = G is continuous, we may assume tha t the m + n observa-

tions are distinct. Since the samples are drawn from the same population, we m a y imagine tha t they were obtained by first drawing ra + n observations, and then select- ing a t random m of these to form the first sample. Thus, under the null hypothesis

/ - - \

F = G , each o f the I r a : n) possible orderings of the samples with respect to each

otherhasthcsameprobabilityl/(m:n).Aswithallrank-ordertests, theproblem of computing P1 and P2 reduces to a purely combinatorial one.

The solution of the combinatorial problem, and incidentally the calculation of the values of D+ and D, is aided by a graphical device. We arrange the m q- n observa- tions in increasing order on a common sequence, and associate with this arrangement a pa th in the x ,y plane. We begin at the origin, and (reading the observations from smallest to largest) take a unit step to the right for each x-observation, and a unit step up for each y-observation. The pa th terminates a t the point (m, n), and from it we can reproduce the ranks of the two samples in their common array. Figure 1 illustrates the process for samples in the order indicated by xyxyxxyyxx.

Such a graphical representation is of course a standard method in classical prob- ability, for example in the problem of gambler 's ruin. I t is usually presented with steps to the right and left instead of to the right and up (Korolyuk and ¥aro§evs 'k i i 1951, Gnedenko and Korolyuk 1951) but the present version, which is used by Drion (1952), is more convenient for use with ordinary graph paper.

470

ARKIV lvOR MATEMATIK. Bd 3 nr 43

J J

J /

J

J J

J /

f

Fig. 1.

Q

J f

f J

J J

J

Suppose that of the first x + y observations in the common array, just x come from the first sample. Then between the (x + y)th and (x + y + 1)st observations, we shall have F,~(x) - G~(x) = x / m - y /n = (nx - mY)Iron, which is proportional to the distance of (x, y) from the diagonal of the rectangle with corners (0,0) and (m, n). Therefore to determine D + (D) we need only locate those points Q+ (Q) on the path

which are farthest below (farthest from) the diagonal. In Figure 1, D ÷ = 61, corre- sponding to either of the points labelled Q÷, while D -~,- 1 corresponding to the point labelled Q.

To compute P~ for an observed path, we may count those paths which never get as far from the diagonal as the farthest point on the observed path. Construct boundaries (Figure 2), parallel to the diagonal and at the same distance from it as Q. The number A (x,y) of ways to go from (0, 0) to (x, y) while staying strictly inside the boundaries satisfies the recursion formula

A (x, y) = A (x - 1, y) + A (x, y - 1) (2.1)

with starting values A (0, y) = A (x, 0) = 1. The desired value A (m, n) can be computed

by simple additions, asillustrated on Figure2, w h e r e P ~ = l - A ( m , n ) / ( m : n ) =

97/105. This technique, which will be referred to as the inside method, is effective for small sample sizes but becomes rapidly less so as m and n increase. For fixed values of P2 the number of additions increases as n '/' and the size of the numbers increases exponentially.

The work can be substantially reduced by two devices: (i) Until the boundary is reached, we are simply generating combinatorials by Pascal's triangle, so that an initial part of the work is unneeessay. (ii) By symmetry, we need carry on only until x + y => ½(m + n), and use the fact that the number of paths through (x,y) is the prod- uct of A (x, y) and A (m - x , n - y ) . Finally, when m and n have a large common factor, and P~ is large, Polya's (1948) method for exact sequential analysis may be. useful.

471'.

J. L. HODGES, JR. , The Smirnov two-sample tes t

This inside method can also be used for P1, but since the number of additions now increases as n ~, the alternative outside method is usually preferable. We count the paths which reach the (lower) boundary, classifying them according to the point (x,y) a t which they first reach (or pass) it. I f B(x,y) is the number of ways to go from (0, 0) to x,y) without previously reaching the boundary, then the to ta l number of paths reaching the boundary is

~ B ( x ' Y ) ( m + n - x - y ) y n - y (2.2)

W e m a y eomputeBsuccessivelyfory=O, 1,...,byobservingthatB(x,y)is(X+y y)

diminished by the number of those ways of going from (0,0) to x,y)which previously reach the boundary. The lat ter number can easily be found with the aid of earlier B values. This process, proposed by Korolyuk (1955a), is illustrated below for the problem of Figure 2.

f

J I

J J

l J

2

I 2

!Jl f

f .... s

L I 4

j J

J 2 f

J

Fig. 2.

8 16

J

j *

J f

(x+y) B(x,y) (re+n-x-y) y x ~ y l \ n-y

0 2 1 3 10 1 70 1 4 5 2 2 l 0 2 5 21 7 3

Thus, B ( 4 , 1 ) = 5 - 3 , where the 3 represents the number (~) of ways to go from

(2,0) to (4,1). The sum of products of the final column is 111, the number of paths reaching the boundary, so tha t P1 = 111/210.

We note tha t the outside method can also be used for P~ when the boundaries are so far apar t tha t a pa th cannot reach both, in which case P2 = 2P1- Even if a few paths can reach both boundaries, i t may be best to allow for these separately and use the outside method.

472

ARKIV FOR MATEMATIK. B d 3 nr 43

(b) T a b l e s

Massey has publ ished two tables of P~, bo th computed by the inside method. (i) I n Massey (1951) we are given 1 - P 2 , t o f rom 2 to 6 significant figures, for

m = n = 1 (1)40 and for a = 2 (1)13 where d = a / n . At m = n = 40 the last decimal is no t reliable. This table is a useful complement to formula (2.4) below.

(ii) Massey (1952) gives 1 - P2 to 5 decimals, for all n < m < 10 and all d, and for selected values of n < m and d for m > 10. Unfor tuna te ly this table does no t seem to be reliable. I n checking abou t 125 values, using the method of Section 4, I found the following 16 instances in which I could no t ver i fy Massey's value of P2- The first of the given values is taken f rom Massey (1952), while the second is mine.

T a b l e 1.

, n m d P r ( D ~ _ d ) n m d P r ( D ~ _ d )

10

36 21 26 27 28 35 50 53

00874 00816 00533 00466 34297 31313 28205 25221 22238 19254 06371 05594 01584 01399 01100 00946

7 8

9

12

15

10 10

10

16

20

56 22 23 54 63 30 31 31

00607 00452 09877 09511 07683 07043 03436 03027 00719 00704 00577 00525 00376 00331 01414 01363

I t seems unlikely t h a t Massey's second table will be ex tended in its present form to cover very large sample sizes, because the number of possible values of D increases so rapidly. I t can be shown tha t D has 1 + [½ r s] + (t - 1) r s possible values, where m = r t, n = s t , and r and s are relat ively prime. While 568 entires suffice for all n < m _< 10, 8707 would be required for n < m < 20, and one can show (using known facts on the densi ty of relative primes) t h a t the number of entries required to cover n < m < M is asymptot ica l ly M412~(3) - $ ( 4 ) ] / 1 6 ~ ( 2 ) = 0.0502 M ~. While P1 and Pz could be p rogrammed for efficient electronic computa t ion , the cost of publishing an adequate table would be excessive.

(c) C l o s e d e x p r e s s i o n s

I n a few special cases one can obta in expressions for P1 and Pz which are relat ively easy to compute.

(i) When m = n our rectangle becomes a square, the boundaries have slope 1, and we can use a reflectional method . Le t the b o u n d a r y be y = x - a , where a = 1, 2, ..., m. A pa th reaching the b o u n d a r y will first do so a t some point Q. We reflect t h a t pa r t of the pa th from (0,0) to @ a b o u t the boundary , generat ing a pa th f rom

( a , - a) to (n, n). As the number of these is ( 2 n t , and as the correspondence is one- \ ~ b - - a /

to-one, we have

4 7 3

J. L. HODGES, JR., The Smirnov two-sample t e s t

To compute P2, we must allow for paths which reach both boundaries, which can be done by repeated reflections. We find

(2.4)

The argument is classical, and is given for example by Gnedenko and Korolyuk (1951) and Drion (1952).

(if) I t has recently been pointed out tha t similar results hold when m = n p, where p is a positive integer. Recall the quant i ty B entering into the inside method. Korolyuk (1955a) noted tha t of those paths to the point (x,y) on the boundary p y = x - a which have previously reached it, exactly 1/(p + 1) approach (x,y) through (x,y - 1). From this it follows tha t

B ( x , y ) = ( x ; Y ) - ( p + l ) ( x ; Y - 1 1 )"

When this is substi tuted into (2.2) we see tha t the number of paths reaching the boundary is

u=0 ( P + I ) y + a \ y n - y

We then of eourse have P l=N (a) / ((P+nl )n ) .

Two a t tempts have been made to obtain a corresponding result for P2. Korolyuk (1955a, p. 86) produced a formula, but it contains a parti t ional sum tha t would be difficult to compute. The formula given in Blackman (1956) is unfortunately in- correct, and the revised formula (Blackman 1957) is not suited for easy computation.

(iii) A closed formula (4.4) for P1 when m = n + 1 is developed in Section 4 below.

(d) Large-sample approximations (i) In his original papers (1939a, b) Smirnov proved that, as m and n - + o o so tha t

m/n--~q we have, for fixed z > 0,

Pr{Vmen+n D+>=z} -+e-2z'' (2.5)

P r { /m~ n+ nD > Z} ---> l - K (z) -- 2 [e-2~' -e-2(2~)~ + e-2¢3")' - ... ]. (2.6)

The function K, which also appears in the limit theory of the Kolmogorov test, has been tabled by Smirnov (1939 b, 1948) to 6 decimals for its argument at intervals of 0.01 over the entire range. Alternative proofs or heuristic proofs of (2.5) and 2.6) have been given by Feller (1948), and Doob (1949). (See also Donsker 1952.)

Smirnov's theorems suggest the introduction of new random variables Z = Umn/(m + n)D and z + = Vmn/(m + n)D +. We shall hereafter always restrict z

474

ARKIV FSR MATEMATIK. B d 3 n r 4 3

to possible vMues of Z and Z+, and shM1 refer to z as the distance variable of the boundary. I t is clear that, if z' and z" are consecutive possible values of Z or Z + and if z' > z => z", then P r {z (+) => Z} = P r {Z (+) _-> z"), so we lose no generality by re- stricting z. To permit z to assume arbi trary values, as is customary in Che literature, leads to considerable and needless notational complication. Similarly, d will always denote a possible value of D or D +, with z = Vmn/(m + n) d, etc.

I t is a truism tha t a limit theorem can be used to justify many different large-sample approximations. For example, we might on the basis of (2.5) approximate Pr (D+ >= d) by exp { - 2mnd2/(m + n)}, but this quant i ty could with equal reason be used to approximate Pr(D+> d), which in some cases is substantially different.

I t has been noted by Drion (1952) tha t good results are obtained, when m = n, if we approximate P2 = Pr (D >= d) by the quant i ty 1 - K ( } / ~ d). The corresponding observation for P1 has been made by H. E. Daniels (J. Roy. Stat. Soc. B (1956), V. 18, p. 22). These observations suggest the use of the large-sample approximations

t5 l = e -e~' and t 3 2 = l - K ( z ) ,

for P1 and P2 respectively. This numerical finding can be reinforced by an asymptot ic expansion of (2.3) and (2.4). Using Stirling's formula and expanding the logarithms involved, i t is easy to show tha t when n-->c~ and a = 0 (}/n),

{a2 2 o4 (1)) - - - ~ ~ - 0 • P l = e x p n 2 n 2 6 n 3

The substitution a = z 2}/~ gives

which shows tha t the use of/31 as an approximation for P1 will lead to a relative error of order I/n, when m = n. Note tha t in this case, the customary continuity correction would introduce an error of order 1/Vnn; but see Section 5(a). A similar analysis shows tha t

P2=e-2~'-e-~(2z)~+e-~(3~)~-"'+O = P 1 - P ~ + P ~ . . . . + 0 n " (2.8)

Essentially these expansions are to be found in Gnedenko (1952). (The version in Gnedenko (1954) appears to be in error.)

(ii) Korolyuk (1954, 1955b) has recently developed asymptot ic expansions for P1 and P2 giving the terms of order 1/~/n and 1/n for arbirary m, n. His formulae would appear to provide means for dealing with the practical problem of determining P1 and P2. However, their correctness has been challenged by Blackman, in his review of Korolyuk 's paper [Mathematical Rewiews 16 (1955) 839], and again in Blackman (1956). He finds tha t Korolyuk's results are not consistent with earlier results of Smirnov (1944) on the one-sample problem, with some of Gnedenko's results for m = n, nor with Blackman's own results for m = np .

475

J . L. HODGES, JR. , The Smirnov two-sample t e s t

As a consequence of the theory developed in Section 4, it follows that

I/?z [e~ z'm.- P r { z + > ~ . n } - 1]

does not tend to a limit as m, n-->oo, Zm.n--->Z, m/n---->l. Thus, not only is the particular expression for PI given by Korolyuk wrong, but no general expression of this simple character can be right. Korolyuk uses analytic machinery on a function of two discrete variables in a formal way, without verifying its applicability. In particular he ignores the lattice structure of the boundary of his region, and it fol- lows from the development of Section 4 that the structure of the boundary in some cases influences the term of order 1//l/nn.

3. Numerical examination of the Smirnov approximations

The preceding section leads to this main conclusion: except for quite small m, we shall usually either have to carry out a rather heavy computation to obtain the signifi- cance probablility, or else rely on the approximations ~51 and/52 based on Smirnov's principal term hmit theorem. Although these have been in use for nearly twenty years, I have not been able to find any numerical examination of their accuracy, except when m = n. The present section presents the results of a brief numerical study, for n = 12, m = 13(1)18, and 0.05> P~> 0.002.

We shall report our results in terms of the relative, not the absolute, accuracy of the approximations. This may be motivated from the point of view of the Neyman- Pearson theory, by observing that an error of given percentage in determining the significance level of a test will usually result in a percentage error of the same order of magnitude in the power fucntion generally. Again, from the Bayesian viewpoint, a percentage error in small P results in a posteriori odds in error by about the same percentage, without regard to the absolute error in P.

In the range studied,/~2//P2 and/~1//PI will be nearly identical. From (2.6) we see that / 5 = 2~51[1 _/5~ +_~ . . . . ] so that the two-tailed approximation is almost exactly double the one-tailed approximation. Correspondingly, the actual value P2 is either exactly 2 P1 or else very nearly so throughout our range. The required values of P1 were computed by the method of Section 4 and by the outside method.

The results are shown as Figure 3, which for each m gives loge(/~2/P2) against z. An examination of this figure reveals several interesting features.

(i) The relative errors are perhaps surprizingly large. For comparison, for m = n = 12,

the relative errors of the two values of P2, in the same range of P2, are 9 % and 23 %. When we add a single observation to one sample, the relative errors are increased to range from 38 % to 180 %. This example may serve as a useful warning against the common belief that increased sample sizes always favor an asymptotic approxi- mation.

(if) The relative error is by no means monotone in z; instead there are wild oscilla- tions at m = 13, which gradually subside as m is increased to 16, but which seem to reappear at m = 17 only to vanish at m = 18.

(iii) I f we t ry to average out the oscillations by some sort of trend line, we see that the "average" relative error increases--perhaps linearly--with z.

476

ARKIV FOR MATEMATIK. Bd 3 nr 43

160

12C

12(

80

40 "120

80

13

1.4

15

12C

4~

12C

16

17

8C

4C

,20 18

80

40

I I • L3 m'.s b ;3 1.5 .

Fig. 3.

(iv) But , for f ixed z, t he average is no t mono tone in m. I n genera l i t falls as m is increased, b u t a t m = 17 i t is h igher t h a n a t m ,-- 16.

These numer ica l resul ts raise bo th p rac t i ca l and m a t h e m a t i c a l problems. The Smi rnov a p p r o x i m a t i o n is seen to be h igh ly inaccu ra t e for va lues of m and n which are a l r e a d y large enough for d i rec t c o m p u t a t i o n s to be arduous . F u r t h e r , some m a t h e m a t i c a l exp l ana t i on is desi rable for the phenomena jus t observed. W e shal l in t he two fol lowing sect ions give pa r t i a l solut ions for these p rob lems .

4. Nearly equal sample sizes

Considera t ions of eff iciency or s y m m e t r y usua l ly l ead to specifying m = n in the des ign of c o m p a r a t i v e exper iments , b u t one or more obse rva t ions is of ten lost . As a resul t , the p rob l em of ca lcula t ing/ )1 and P2 when m and n are n e a r l y b u t no t e xa c t l y equal acquires p rac t i ca l impor tance , especia l ly since the Smi rnov a p p r o x i m a t i o n is p a r t i c u l a r l y bad in th is case. W e shall now develop a t h e o r y a p p r o p r i a t e when m - n = c is small . W e begin b y es tabl ish ing a genera l lower b o u n d for P1.

F o r b r e v i t y we shal l refer to t he line segments y = x - a, for a = c, c + 1 . . . . . m a n d y => 0, x _< m, as mirrors. E a c h p a t h reaches a t leas t one mirror ; of these, t he

477

z. L. HODGES, Jm, The Smirnov two-sample test

one wi th l a rges t a will be called the end mirror. A p a t h touches i t s end mi r ro r in a t l eas t one poin t ; t he lowest of these is the end point. The r a n d o m coord ina tes of the end p o i n t wil l be deno t ed b y (X, Y) and we wr i te X - Y = A. W e shall use t h e r a n d o m var i ab le A to i ndex the mirrors .

The ref lec t ion m e t h o d (Section 2c) gives a t once

P r ( A ~ a ) ~ ( m + n ] / ( m : n ) • - \ n + a ] / (4.])

F u r t h e r m o r e , i t enables us to calculate t he n u m b e r of ways to go f rom (0,0) to (a + y, y) w i t h o u t p rev ious ly touching the mi r ro r A = a, which will be de no t e d b y I(a,y). This n u m b e r is the same as the number of ways to go f rom (0,0) to (a + y - 1,

( a + 2 y - 1 ) y) w i thou t touch ing A = a. There are ways to reach (a + y - 1, y), b u t

Y

ref lec t ion we see t h a t ( a ÷ 2 y - 1~ of these have touched A = a. Hence b y \ y - 1 /

I ( a , y ) = ( a + 2 y - l ] ( a + 2 y - 1 ) . y ] - \ y - 1

(4.2)

Simi lar ly , the n u m b e r of ways to go from (a - y , y) to (m,n) w i thou t crossing (but poss ib ly touching) t he mir ror A = a is I (n + a + 1 - m, m - a - y). Thus

( m : n ) p r ( A = a a n d Y = < u ) = ~ = o ~ I ( a , y ) I ( n + a + l - m , m - a - y ) . (4.3)

Consider now a lower b o u n d a r y line L, corresponding to a mos t d i s t a n t p o i n t (x +, y+). The l ine L: m (y - y+) = n (x - x +) will in te rsec t a t leas t one mirror ; le t t he h ighes t mi r ro r which L in tersec ts be A = ~. Then ~ is t he smal les t in teger g rea te r t h a n or equal to x+ - m y+/n. I t is easy to show t h a t t he o rd ina t e of the po in t of in te rsec t ion of L a n d A = ~ , which we shall denote b y vn, is equal to n[1 -g (x+-my+/n)] /c , where g (u) denotes t he f rac t ional p a r t of u. The va r i ab le v thus def ined will be called the phase variable of L. The o rd ina te of the p o i n t of in te rsec t ion of L wi th A = ~ + i is t hen n (v + i/c). Fina l ly , for t he d i s tance va r i ab le z we have the express ion

z= r e + n \ m

so t h a t ~=vc+zV (n÷c)(2n+c)

C

A n y p a t h whose end po in t lies on or below L m u s t of course reach or pass L. There- fo re

m n ply_ m+ + ~ ~. I (a ,y ) I ( n + a + l - m , m - a - y ) . (4.4) m ÷ a=~ u=o

478

ARKIV FOR MATEMATIK. B d 3 n r 43

We shall denote the right side of ( 4 . 4 ) b y ( m : n ) P ~ . P ~ i s t h u s a l o w e r b o u n d f o r

P1, and equals, P1 if and only if L cuts a single mirror, which will always be the case when c = 1. Thus (4.4) provides an exact closed expression for P1 when m = n + 1. I t is intuitively plausible tha t P~ should be close to P1 when c is small, and it is shown below tha t P 1 - P ~ = 0 ( l / n ) when n -+c~ and c is bounded.

The most notable feature of P~ is t ha t i t depends in a simple way on a function, I , of only two variables, while P~ itself depends on the four variables m, n, x +, and y+. We give a short table of I , suitable for computing Pa when m_-_ 30. From the 323 values of Table 2, and a table of combinationals, one can easily obtain any of thou- sands of accurate estimates for P1. (As explained in Section 3, 2P* may then be used to est imate P~ if the lat ter is not too large.)

As an example, we take m = 20, n = 16, and seek P1 corresponding to a pa th with farthest point (17,6). The boundary L is 5 ( y - 6) = 4 ( x - 17), which cuts the mirrors A = 10, 11, so tha t a = 10. The ordinates of the points of intersection of L with A = 10 and A =11 are respectively 2 and 6. As most of the mirror A = 11

lies below L, it is quicker to compute the complement of (4.3) with respect to (m : n)

Pr (A = a). We have

(36) p~ = (3:) -[I(11'7) I (8'2)+I (11'8)I (8'1)+I (11'9)I

+ [I (10, 0) I (7, 8) + I (10, 1) I (7,9) + I (10, 2) I (7,10)] = 9.1407 × 107

This gives P~ = 0.01251. Massey (1952) gives P2 = 0.0251I, and since the probabil i ty of reaching both boundaries is negligible, this implies P1 = 0.01256. In the range P 2 < 0.1, 2 P~ never differs from P2 as given by Massey by as much as 1% of P2. The accuracy of P~ will of course decrease as c increases, bu t even at n = 12, m = 20 ~P* is considerably superior t o / 5 i.

We shall now develop an asymptotic expression for P1, correct to terms of order 1 /~n , for what may be called "nearly equal" sample sizes. Consider a sequence (mk,n~) of sample sizes, such tha t nk-+c~ and m k - n k = % is bounded. With each (mk, n~) is associated a boundary L~ with distance variable z~, phase variable vk, and significance probabil i ty Plk. We assume tha t zk is bounded and bounded away from 0, so tha t ~k will be of exact order ~/~. To simplify typography we shall omit the subscript k hereafter.

A straightforward application of Stirling's formula to (4.1) shows tha t

/ P r ( A > _ - a ) = e x p - - - + - - + 0 (4.5) n ~b

when a = 0 (~n). From this it follows that

479

oo

Ta

ble

2.

T

ab

le

of

I (a

, y

).

Th

e su

pe

rsc

rip

t is

th

e

ex

po

ne

nt

of

tha

t p

ow

er

of

10 b

y w

hic

h t

he

e

ntr

y i

s to

b

e m

ult

ipli

ed

.

I(a,

O)

= l

; I(

a, 1

) =

a;

I(l,

y +

l)

=I(

2,y)

. t~

y a

= 2

3

4 5

6 7

8 9

10

11

12

13

14

15

16

17

2 3 4 5 6 7 8 9 10

11

12

13

14

15

16

17

18

19

20

21

22

23

2

4

25

26

27

2

8

5 14

42

13

2

42

9

14

30

4

86

2

1680

1 58

791

20

80

" 7

42

9 ~

2

67

4 a

96

95

a 35

364

12966

47766

17

67

" 6

56

4 ~

2

44

7'

91

48

' 3

43

1"

12

90

~

48

62

~

18

37

lo

69

53

lo

26

37

I~

10

02

x~

9 14

2

8

48

90

16

5 2

97

5

72

1O

01

20

02

3

43

2

70

72

1

19

3 ~

25

191

41

99

x

9044

1 14

92 ~

3

26

9"

53

49

~

1189

" 1

93

2 ~

4

34

6 ~

7

02

0 ~

15

974

2566

4 58

934

9429

4 2

18

3 ~

3

48

06

8

12

06

1

29

0 ~

30

30

e 4

79

7 a

11

34

7

17

90

' 4

25

5'

67

02

' 1

60

1 ~

25

16

8

60

38

8

94

68

s 2

28

2"

3572 ~

8

64

3 ~

13511o 32801o

51171o

12

47

~

1942

11

47

47

~

73

85

ix

20

27

75

11

0

27

5

42

9

1001

1

63

8

36

40

6

18

8

1326

1 23

261

4845

1 87

211

1776

" 32

692

6538

~

1226

3 2

41

4"

46

02

a 89

488

1730

4 33

274

65

13

~

1241

~

2456

6 46

406

9280

5 17

40e

35

12

e 65

416

13

31

' 2

46

5'

50

53

' 93

08

~

19

21

8

35

22

s 7

31

5"

13

35

~

2789

9 50

71"

10

65

l°

1929

10

40

72

lo

73

51

lo

1559

11

2805

11

35

15

4

63

7

25

48

9

99

6

3876

1 14

92"

57

20

" 2

18

7 a

8

35

1 a

31

874

1216

6 46

406

1772

" 6

76

9 e

25

88

' 9

90

4'

37

93

" 14

54"

55

79

g

2142

1°

82

34

lo

44

2

08

9

10

3

80

8

1550

1 62

021

24

52

z

96

14

z 3

74

9"

1457

4 56

454

2183

6 84

366

3257

6 1

25

7'

4851

7 18

72"

7225

8 27

899

10

77

lo

4162

101

54

65

27

3

35

0

12

60

1

70

0

55

08

7

75

2

2326

1 33

921

9593

1 14

422

38

94

" 6

00

9 ~

15

623

24

67

" 6

21

6 e

1002

4 24

584

4032

4 96

774

1613

5 3

79

6 a

64

19

s 14

86"

25

45

e 5

80

2 s

10

06

' 2

26

3'

39

65

' 8

81

5'

1560

8 3

43

2"

6128

8 13

35"

24

05

" 51

949

9426

9 2

02

0 lo

77

44

0

22

44

10

661

4828

1 2

11

5"

90

45

" 37

998

1574

4 64

514

2623

6 10

596

42

55

" 1

70

2'

67

84

' 26

968

10

69

' 42

329

90

5

44

2

90

7

1436

1 67

301

30

36

" 13

32"

57

23

" 24

194

1010

6 4

17

2 s

17

10

~

69

62

" 2

81

9'

1136

" 4

56

4"

1827

~

10

4

66

3

37

05

19

021

9209

1 4

27

6 I

1924

8 8

45

4 s

36

46

' 15

505

6509

5 27

076

11

17

' 4

58

1'

18

68

" 7

58

2"

119

79

8

46

55

24

791

12

40

~

59

20

" 2

73

1"

1227

4 53

994

2336

5 99

756

42

12

" 1

76

2'

73

15

' 3

01

8"

13

5

95

0

57

75

31

881

1644

2 8

07

3 m

3

81

7 a

1753

4 78

684

3466

6 15

046

64466

27

34

' 1

15

0 s

15

2

11

20

7

08

4

4048

1 2

15

3 l

1086

3 52

598

2468

4 11

306

50

67

s 2

23

5 e

9721

6 41

807

17

0

13

09

8

60

2

5083

1 27

852

1442

8 71

528

3430

4 1

60

0 b

73

065

3275

6 1

44

6'

18

189

15

18

10

351

6318

1 ~

" 35

632

1893

3 9

61

2 a

4707

4 2

23

9 s

10

40

e 4

73

9 e

ARKIV FOR MATEMATIK. B d 3 nr 4 3

On substituting a = V~2--n z + vc + 0 ( 1 / ~ n ) i n t o (4.5), we find

P r ( A > ~ z + c ) = e x p - 2 z ~ - ( 2 v c + c ) ~ n + 0 •

Similarly

P r ( A = ~ + i ) = 2 V 2 Z e - ~ ' + O ( 1 ) f o r i = O , 1 , - - . . . , c . Vn

(4.6)

(4.7)

Since the mirrors cut b y L have probabili ty of order 1/Vn, it is enough to deter- mine pr(y<__ u n l A =o~+i) to constant order. From (4.2), it can be shown tha t

I (a, y) = 2 ~ y+a-1 a exp {-- a2 (4.8)

when y = 0 (n). When we substitute this into (4.3) and make the usual integral ap- proximation of a sum, we find

(2 u - 1)] Pr(r<un[ + 0(1), (4.9)

where (I) is the normal distribution function. On substituting (4.6), (4.7), and (4.9) into (4.4), we obtain

P ~ = exp - 2z z - Vn ~ , f f i 0 ~ tFz v + (4.10)

[z(2u- where

I t remains to show tha t P1 -P~ = 0 ( l /n ) in order to establish (4.10) as an expres- sion for P1 with error of order 1/n. The argument will be given for simplicity only in the case m = n + 2. A path counted in P1 but not in P~ must (i) reach but not pass the mirror A = a for x + y < n, and then (if) reach but not pass the mirror A = a + 1 for x + y > n. Let S be the point in which such a pa th crosses x + y = n. For a given value of S, the behavior of the path after S is conditionally independent of its behavior before S. As a consequence of (4.6), each of the events (i) and (if) has probabil i ty of order 1/Vnn, uniformly in S.

The expression (4.10) has a number of interesting features. (i) Each of the three terms in the exponent has an interpretation. Thus - 2 z 2 corresponds to the Smirnov approximation P1. The second term - V2 z/Vnn represents the "average" error of the Smirnov approximation; we note that it is of order 1/Vn, and that it is proport ional to z, as was suggested empirically in section 3. Further, it does not depend on c. The third term represents the oscillatory component. This is also of order 1/~/n, which

shows that i t is not possible to express P1 to accuracy 1 /~n in a formula of the type proposed by Korolyuk.

481

J . L. H O D G E S , J R . , T h e S m i r n o v t w o - s a m p l e tes t

(ii) The function ~F was derived by Malmquist (1954), using the heuristic argument of Doob (1949), as the limiting conditional distribution of the ordinate of the furthest point, say Y+, given the value of Z +. Our argument, which is restricted to the case when c is bounded, permits this limiting distribution to be derived rigorously when Z+ is restricted to an appropriate interval, and also shows that the result is not correct for arbitrary small intervals for Z +. We thus have an example of a problem in which the heuristic argument fails unless properly restricted.

C--1

(iii) From symmetry we see that ~d~ (u) + ~F~ (1 - u) = 0. I t follows that ~ ~F~ i=0

l/c

v + c = 0 for 2 cv aninteger, a n d t h a t Y. | v + - | d v = O , so that the oscillatory ~=o \ c /

o

part averages out to zero over a cycle of the phase variable v. Further, it can be

shown from the Euler-Melaurin formnla tha t max ~ ~I~ v + --~0asc-~oo, whieh

"explains" the phenomenon, observed in Section 3, of damped oscillations as m is increased. A few values of the maximum oscillation are given in Table 3.

T a b l e 3. m a x t ~ 0~Fz(x+~) .

z c = l 2 3

1 0.13 0.04 0.006 1.5 0.21 0.03 0.013 2 0.26 0.10 0.011

(iv) The formula shows that the error of the Smirnov approximation may be quite large even for substantial values of n, particularly when c = 1. For example, when

m = n + 1 and z = 1.5 corresponding to P~ = 0.022, the relative error of P~ at the least favorable phasing is about 3/Vnn, so that the sample sizes must be about 900 to assure a 10 % relative error. The same goal is achieved at n = 20 when the sample sizes are equal.

We conclude with an extension of these results to the two-sided test. As remarked above, in practice the approximation P2 = 2 P1 will usually serve; and since the re- sults for P~ are considerably more complicated than those for P1, we shall give only an outline of methods. Corresponding to the end mirror A, we now have upper and lower end mirrors, say A 1 and Au. Instead of (4.1) we now obtain, analogously to (2.4),

n ) r ( A ~ > = a o r A l ~ a - c ) - - . = \ n ÷ a ] \ n + 2 a ] \ n ÷ 2 a - c ] ÷ ' ' "

(4.11) The first term on the right side of (4.4) is thus replaced by (writing~z + c for a in (4.11),

m + n

\ n + 2 ¢ ¢ + c ] "

482

ABKIV FhR MATEMATIK. Bd 3 nr 43

I f we carry out work analogous t o t h a t which gave (4.6), we find

P r ( A l <=oc or A ~ > o ~ + c ) = 2 P r ( A > = o ~ + c ) { 1 - [ P r ( A > = o ~ + c ) ] a + . - ' } (4.12)

a relation analogous to the f o r m u l a / 5 = 2/51(1 - ~ha + ...) of Section 2. I f n or P2 is ve ry large, the second te rm in (4.12) m a y be wor th using.

A similar extension is possible for the second te rm on the r ight side of (4.4). Ins tead of the funct ion I , we h a v e J defined by J ( a , y ) = the number of ways to go from (0,0) to (a + y , y ) without previously touching either A = a or A = - a. A n application of the reflection method yields

J (a,y) = I (a ,y) - l ( 3 a , y - a ) + I (ha, y - 2a) - I (T a, y - 3a) + .. .

which enables one to compute an analog to P~. Corresponding to (4.8), J (a, y) has an expansion consisting of the r ight side of (4.8) multiplied by 1 - 3 e -2 a,/~+ ....

I f we ignore pa ths influencing P~ only to order I / n , we can classify the paths reaching one or bo th boundaries into the following sets. (a) Pa ths reaching A = ~ + c or A = - ~ or both; (b~) paths reaching A = ~ + i below its intersection with the lower boundary , bu t not reaching A = ~ + i + 1 nor A = - ~; (c~) a set of pa ths analo- gous to b~ bu t with the boundaries interchanged. The set (b~) will contr ibute to P2 a quan t i ty analogous to (4.9), bu t with (4.9) replaced by a series whose first t e rm is

(4.9). The remaining terms will be of order 1/Vnn, bu t in practice they are usually negligibly small.

5. Condu~on

We conclude with three remarks or conjectures on the na ture of the general problem

of comput ing P1 to order 1/~nn. (a) I t seems likely t h a t the oscillatory behaviour demons t ra ted above for m / n

near 1, will also obtain for m / n near any rational number r / s , where r > s and r and s are relat ively prime. We shall give an heuristic a rgument which also yields a quant i ta- t ive formula for the oscillation. I f the lat ter is correct, the oscillation will be of prac- tical interest in only a few cases.

Suppose m is "nea r" (r / s ) n. Analogously to the end mirrors of section 4, we define the end l ine of a pa th as the line r (y - y+) = s (x - x+), and again index these lines by their x-intercept, a. Now s a takes on consecutive integral values, and consecutive z

values differ by 1 / V n ~ + s) . This suggests t h a t the limiting probabi l i ty of an end

line is 4 z e - e Z ~ / V ~ r ( r + s) . This is Y 2 / r ( r + s) t imes its value when m = n. We define the end point of a pa t h as the lowest (highest) point which the pa th

reaches on its end line, when m > ( < ) r n / s . Malmquist ' s heuristic a rgument suggests t h a t the limiting conditional distr ibution of the ordinate of the end point , given the end line, is again given by (4.9); in fact, any other form for this dis tr ibut ion would no t be reconcilable with the result of Malmquist cited above.

Finally, we notice t h a t the number of end lines cut b y the b o u n d a r y is, in the limit, (i) c s when m = r n / s ++_ c, or (ii) c r when n = s m / r + c. Combining, we should

find the asymptot ic magni tude of the oscillation to be ~ 2 / r (r + s) t imes as ~oTeat as it is when m = n + c s in case (i), or m = n + c r in case (ii). I n part icular, the oscilla-

483

J . L. HODGES, J R . , The Smirnov two-sample test

t ion at (n = 12, m = 17) should be about V2/15 = 0.37 times as great as t h a t at (n = 12, m = 14). The empirical results of Section 3 are in reasonable agreement with this prediction.

A compar ison of the conclusion of this heuristic a rgument with Table 3 suggests t ha t the oscillatory aspect of P1 is likely to be of practical interest only when m and n are near ly equal, and to a lesser extent when m / n is near ~. Thus, for m / n near ~,

the oscillations should be a t worst only ~/~ (4 + 3) = 0.27 as gTeat as at m = n + 3; or

for z near 1.5, t h e y should contr ibute a t worst 0.015/Vnn to the relative error. Even

near ~, the m a x i m u m oscillation is a t worst about 0 .05/Vn. _ (b) We now ignore the oscillations, and examine the "average" value of P1, say P1, for m and n near ly equal. F rom (4.10) we have

for a ny c > 0. On the other hand, for c = 0, we have from (2.7)

P l = ' l e x p { - 2 z2 + 0 ( 1 ) } •

We are confronted with a disconcert ing discontinuity in P l as a funct ion of c. I t is of interest to note t h a t the two formulae are reconciled if a cont inui ty correction is used.

I n fact, for c > 0, the probabilities of individual z-values are of order 1In '1', so tha t (5.1) continues to hold when a cont inui ty correction is employed. A t c = 0, on the other hand, Pr(Z+ = z) = (2V2z/l/-nn) e -2z' . The use of a cont inui ty correction implies increasing the est imate by half of this amount , so tha t (5.1) used with the

correction will yield, to order l / n , the value/51 . (Incidentally, the oscillations for m = n + 1 are made considerably more regular when the cont inui ty correction is used.)

(c) Since (5.1) holds for any fixed c, it is t empt ing to conjecture tha t this formula m a y be used for a rb i t ra ry values of m and n. This conjecture, however, is easily disproved by considering the case m = n p, where p is an integer. By use of Stirling's formula it can be shown t h a t the y th term of the sum given in Section 2 for N (a) is

Tb~2 7~ba

a l/n exp (n - y) 2 (p + I) y (n - y) + ~/2 z tp ( p + i) ya ( n _ y) 2 p ( p + l ) y

( 2 y - n ) a . ( 2 p + l ) n ( n - 2 y ) a 3 ( 1 ) } + 2 p y ( n - y ) ~- 6 [ p ( p + 1 ) y ( n - y ) ] 2+ 0

when y and n - y are bo th of order n. If we make the subst i tut ions y = v n and a =

] /p-~ + 1) z, we find 1

z exp - 1 - v (1 - v) P1 - 1/2,~,,3 (1_ v) 2v ( i - , , ) 2 Vv(~-+l)~ 0

(p+ 1 ) ( 2 v - 1 ) ( 2 p + 1) (1-2v)z2]} v ( l - v ) ~ v~-(~-- ~2 ] dv .

484

ARKIV FOR MATI~MATIK. B d 3 n r 43

T h e v a r i o u s i n t e g r a l s m a y b e r e d u c e d b y s u b s t i t u t i n g 1 - 2 v = u w h e n 0 < v < ½ a n d 2 v - 1 = u w h e n ½ < v < 1, a d d i n g t h e i n t e g r a n d s , a n d t r a n s f o r m i n g t o t h e g a m m a i n t e g r a l . W e f i n d

3 Y p ( p + 1) V~n ~ 0 (5 .2)

w h i c h a g r e e s w i t h t h e a p p r o p r i a t e spec i a l case of C o r a l l a r y 2 i n B l a c k m a n (1956) a f t e r m a k i n g t h r e e c h a n g e s , a p p a r e n t l y m i s p r i n t s , i n t h e l a t e r .

W h e n a c o n t i n u i t y c o r r e c t i o n is e m p l o y e d , f o r m u l a (5.2) b e c o m e s

T h i s s i m p l e f o r m u l a is t h u s s h o w n t o b e c o r r e c t for m = n, fo r m a n i n t e g r a l m u l t i p l e of n , a n d " o n t h e a v e r a g e " f o r m n e a r l y e q u a l to n . O n t h e b a s i s of a l i m i t e d n u m e r i e £ 1 i n v e s t i g a t i o n , i t a p p e a r s t o w o r k r e a s o n a b l y wel l i n o t h e r eases , a n d c a n p e r h a p s b e r e c o m m e n d e d as a g e n e r a l i n t e r p o l a t i o n f o r m u l a .

University o] Cali/ornia, Berkeley, Cal., U.S.A.

R E F E R E N C E S

BLACK~-N, JEROME (1956), "An extension of the Kolmogorov dis t r ibut ion". Arm. lVlath. Statistics 27, 513-520.

- - - - (1957), "Correction to 'An extension of the Kolmogorov d is t r ibu t ion ' " , unpublished. DO~SKER, MONROE D. (1952), "Just i f icat ion and extension of Doob's heuristic approach to the

Kolmogorov-Smimov theorems". Ann. Math. Statistics 23, 277-281. ]DOOR, J. L. (1949), "Heurist ic approach to the Kolmogorov-Smirnov theorems". Ann. Math.

Statistics 20, 393-403. DRIO~, E. F. (1952), "Some distribution-free tests for the difference between two empirical

cumulat ive distr ibut ion functions". Ann. Math. Statistics 23, 563-574. FELLER, W. (1948), "On the Kolmogorov-Smirnov limit theorems for empirical distr ibutions".

Arm. Math. Statistics 19, 177-189. GNEDE~KO, B. V. (1952), "Some results on the max imum discrepancy between two empirical

distr ibutions". Doklady Akad. Nauk. SSSR 82, 661-663. G~EDENKO, B. (1954), "Kri ter ien fiir die Unver~nderl ichkeit der Wahrschcinlichkeitsvertei lung

von zwei unabh~ngigen Stichprobenreihen". Math. Nachr. 12, 29-66. GNEDENKO, ]B. V., and Kom)LYU~, V. S. (1951), "On the max imum discrepancy between two

empirical distr ibutions". Doklady Akad. Nauk SSSR 80, 525-528. GNEDENKO, B. V., and RVA6EVA, E. L. (1952), "On a problem of comparison of two empirical

distr ibutions". Doklady Akad. Nauk SSSR 82, 513-516. GNEDENKO, B. V., and STun~Ev, Yv. P. (1952), "Comparison of the effectiveness of several

methods of testing homogeneity of statistical material" . Dopovidi Akad. Nauk Ukrain. RSR, pp. 359-363.

KOLMOOO~OFF, A. (1933), "Sul la determinazione cmpirica di une leggi di distribuzione". Giorn. Ist. I tal . At tuar i 4, 83-91.

KOLMOOOROV, A. N., and Hr~6IN, A. YA (1951), "The work of N. V. Smirnov on the inves t iga t ion of properties of variat ional series and on nonparametr ie problems of mathemat ica l statist ics". Uspehi Matem. Nauk 6, no. 4 (44), p. 190-192.

Ko~oLYuK, V. S. (1954), "Asymptot ic expansions for A. N. Kolmogorov's and ~q. V. Smirnov 's criteria of fi t ." ])oklady Akad. Nauk SSSR 95, 443-446.

- - - - (1955 a), "On the discrepancy of empiric distributions for the case of two independen£ samples". Izvestiya Akad. Nauk SSSR. Ser. Mat. 19, 81-96.

- - (1955 b), "Asymptot ic expansions for the criteria of fit of A. N. Kolmogorov and N. V. Smirnov". Izvestiya Akad. Nauk SSSR. Ser. Mat. 19, ]03-124.

34t 4 8 5

J . L. HODGES, JR., The Smirnov two-sample test

KO~OLY~, V. S., and Y.~mo~.vs~-~, B. I. (1951), "S tudy of the m a x i m u m discrepancy of two empirical distr ibutions". Dopovidi Akad. Nauk Ukrain RSR, pp. 243-247.

MALMQUIST, STEN (1954), "On certain confidence contours for distribution funct ions", Ann. Math. Statist ics 25, 523-533.

MASSEY, F. J. JR. (1950), "A note on the power of a non-parametr ic test" . Ann. Math. Statistics 21, 440-443.

MASSEY, Fma_,~x J. , J~. (1951), "The distribution of the m a x i m u m deviation between two sample cumula t ive step functions". Ann. Math. Statistics 22, 125-128.

- - (1952), "Dis t r ibut ion table for the deviation between two sample eumulat ives" . Ann. Math. Statist ics 23, 435-441.

POLY.~, GEORGE (1948), " E x a c t formulas in the sequential analysis of a t t r ibutes" . Univ. California Publ. Math. 1, 229-239.

RVA6EVA, E. L. (1952), "On the m a x i m u m discrepancy between two emprirical distr ibutions". Ukrain. Mat. ~urna l 4, 373-392.

SMraNO~F, N. (1939), "Sur les ~carts de la courbe de distr ibution empirique". Mat. Sbornik 6 (48), p. 3-26.

S ~ N O V , N. (1939), "On the est imation of the discrepancy between empirical curves of distribu- t ion for two independent samples". Bull. Math. Univ. Moseou 2:2.

- - - - (1948), "Table for es t imat ing the goodness of fit of empirical distr ibutions". Ann. Math. Statistics 19, 279-281.

VAN DER WAE~D~.N, B. L. (1953), "Order tes ts for the two-sample problem. I I " . Indagat iones Math. 15, 303-310.

Tryckt den 17 december 1957

Uppsala 1957. AImqvist & "Wiksel|s Boktryckeri AB

4 8 6