Refining Bootstrap Simultaneous Confidence Sets

By

Rudolf Beran*

Technical Report No. 139January 1988

*Research supported by National Science Foundation Grant DMS-87-01426.

Department of StatisticsUniversity of CaliforniaBerkeley, California

Refining Bootstrap Simultaneous Confidence Sets

By

Rudolf Beran*Department of StatisticsUniversity of CalifomiaBerkeley, CA 94720

SUMMARY

Simultaneous confidence sets for a collection of parametric functions may be con-structed in several different ways. These ways include: (a) the exact pivotal methbdwhich underlies Tukey's (1953) and Scheff6's (1953) simultaneous confidence intervalsfor linear parametric functions in the normal linear model; (b) the method of asymp-totic pivots, which is an approximate extension of the pivotal method; (c) the methodof bootstrapped roots developed in Beran (1988).

These three methods share several features. Each method simultaneously asserts acollection of confidence sets, one confidence set for every parametric function ofinterest. Each method obtains the uth constituent confidence set by referring a root toa critical value; the uth root is a function of the sample and of the uth parametric func-tion. Each method has the same aim: to control the overall level of the simultaneousconfidence set and to keep equal the marginal levels of the individual confidence state-ments which make up the simultaneous confidence set.

The three methods differ in how they construct the necessary critical values. Thepivotal method requires that the roots be identically distrbuted and that the supremumover all the roots be a pivot- a function of the sample and of the unknown parameterwhose distribution is completely known. The method of asymptotic pivots relaxesthese assumptions slightly by requiring them to hold only asymptotically. Neverthe-less, the practical scope of both methods is very narrow. In contrast, the method ofbootstrapped roots, or B-method, is widely applicable. Under general conditions, itgenerates simultaneous confidence sets which, asymptotically, are balanced and havecorrect overall coverage probability. Moreover, the B-method has a direct MonteCarlo approximation which makes it easy to use. Under supplementary conditions, the

* Research supported by National Science Foundation Grant DMS-87-01426.

- 2 -

pivotal method turns out to be a special case of the B-method.

At finite sample sizes, a B-method simultaneous confidence set may suffer error inits overall coverage probability and lack of balance among the marginal coverage pro-babilities of its constituent confidence sets. Of what orders are these errors? Whatcan be done to reduce them? We answer both questions in this paper, by introducingand studying the B2-method - a double iteration of the B-method. Under regulartyconditions, the B2-method reduces the asymptotic order of imbalance in the B-methodsimultaneous confidence set; and at the same time, it reduces the asymptotic order oferror in overall coverage probability. The B2-method has a direct Monte Carloapproximation which makes its use straightforward, though computer intensive.

KEY WORDS: Simultaneous confidence set; Balance; Coverage probability; Higher-order asymptotics; Double bootstrap

1. INTRODUCTIONIn constructing an ordinary confidence set, the first goal is controlling level. When

constructing a simultaneous confidence set, the initial goals are two-fold: (a) control-ling the overall level; (b) controlling the relative levels of the individual confidencestatements which make up the simultaneous confidence set. Methods for accomplish-ing both aims (a) and (b) are the topic of this paper.

Let xn be a sample of size n whose distribution P0m belongs to a specifiedparametric family of distributions. The unknown parameter 0, which can be finite orinfinite dimensional, is restricted to a parameter space 0. Let T be a parametric func-tion, defined on 0, which has components labelled by an index set U. In other words,T (9) = (T. (9): u e U), where the parametric function T. is the uth component of T.The index set U can be finite or infinite, as will be illustrated in example 1 below.

Suppose that, for each value of u, C,,u is a confidence set for the uth componentT. (0). By simultaneously asserting the confidence sets (C....: u e U), we obtain asimultaneous confidence set Cn for the family of parametric functions T (O) = (Tu(0))The problem is to devise the component confidence sets (C,.) in such a way that

(a) the level of Cn as a confidence set for T (9) is 1 - a;

(b) the level of C,, as a confidence set for Tu (9) does not vary with u.

Property (b), which we term balance, ensures that the simultaneous confidence settreats each component Tu (9) fairly.

We will review three related approaches to this problem and will propose a fourthmethod, which offers better control of overall level and of balance at moderate to largesample sizes. The first approach is the exact pivotal method which underlies Tukey's(1953) and Scheffe's (1953) simultaneous confidence sets in the normal linear model.The second method is the obvious asymptotic extension of the pivotal method.Though they are important, these two approaches tum out to have limited scope,because they require more structure than exists in many situations where we seeksimultaneous confidence sets.

The third approach is the method of bootstrapped roots, or B-method, which isdescribed in sections 1.3 and 2.1. The B-method is widely applicable and generatessimultaneous confidence sets which, asymptotically, are balanced and have correctoverall coverage probability (Beran, 1988). The fourth technique is the method ofdoubly bootstrapped roots, or B2-method, which is introduced in sections 1.4 and 2.2.The B2-method is basically a double iteration of the B-method. Under regularity con-ditions discussed in section 3, the B2-method reduces the asymptotic order of

- 2 -

imbalance in the B-method simultaneous confidence set; and at the same time, itreduces the asymptotic order of error in overall coverage probability. In many cir-cumstances, the exact pivotal method tums out to be a special case of both the B-method and the B2-method. More generally, the B2-method requires a doublebootstrap Monte Carlo algorithm which uses roughly the square of the computer timeneeded by the B-method. A SUN 3/140 workstation carried out the numerical exam-ple of the B2-method reported in section 2.4.

1.1. The Pivotal Method

Let Ru = R,u (xn, Tu (0)) be a confidence set root for Tu (0) - a function of thesample and of the parametric function Tu (0). Suppose that the roots (R.u: u e U) areidentically distributed and that their supremum is a continuous pivot; that is, the distri-bution of supuRu is continuous and does not depend upon the unknown 0. Let Tdenote the set of all possible values of T (0) = (Tu (0)) as 0 varies over the parameterspace. Every point t in this range T can be written in.component form t = (t,, wheretu lies in the range Tu of Tu (0).

Consider a confidence set for Tu (0) of the form

Cnu= ne (Xntu) (1.1)

Simultaneously asserting the confidence sets (C,)u) generates the following simultane-ous confidence set for T (0):

Cn = (t T : ku(xptu cu foreveryu e U). (1.2)Suppose the critical value cU is the largest (1 - a)th quantile of the distribution ofsupuR,u. Then C. has exact level 1 - a, because supuR u is a continuous pivot; andCn is balanced, because the roots (R,u are identically distributed.

Example 1. Consider the classical normal linear model, in which the sample xn hasa N (AP, ca2 I) distribution, 1 is r-dimensional, the regression matrix A has rank r, and Iis the identity matrix. The unknown parameter 0 = (1, a2) has the usual estimateon = (nt n) from least squares theory. Suppose that Tu (0) is a linear combination

TU(0) = U'J3, (1.3)where u is an r-dimensional vector, and that the corresponding root is

R,u = IU'(On - 1) I/ n,u (1.4)where r = u' (A'A)-1 U . The roots (Rn,uI are identically distributed, each havinga t-distribution with n - r degrees of freedom folded over at the origin.

- 3 -

It remains to specify U, the set of r-dimensional vectors over which u is allowed torange. Consider two special cases:

(a) U is a subspace of dimension q. Then supuR,,u is a continuous pivot, distri-buted as (qW)112, where W has an F-distribution with q and n - r degrees of freedom(Miller, 1966, chapter 2, section 2). The pivotal method's confidence set C. is justScheffe's (1953) simultaneous confidence intervals for the linear combinations(u'3: u e U).

(b) U is all pairwise contrasts in a balanced one-way layout. The parameter J isjust the vector of means in this specialization of the linear model. Then sup,,uR is acontinuous pivot, distributed as 21r/2 R, where R has the studentized range distributionwith parameters r and n - r (Miller, 1966, chapter 2, section 1). Now the pivotalmethod yields Tukey's (1953) simultaneous confidence intervals -for all pairwisedifferences in means.

1.2. The Method of Asymptotic Pivots

An obvious approximate extension of the pivotal method is possible when only theasymptotic distributions of the roots (Rk) are identical; and when the asymptotic. dis-tribution of sup.R ,, is continuous and does not depend on 0. In (1.2), take the criti-cal value cu to be the largest (1 - a)th quantile of the asymptotic distribution ofsupR,,u. Then, asymptotically, the simultaneous confidence set Cn is balanced andhas overall coverage probability 1 - a. Both statements are true pointwise in 0. Con-vergence of overall and component levels may or may not occur; see section 4 forfurther discussion of this point.

Example 2. Consider the following nonparametric generalization of the normallinear model in Example 1. The sample xn is distributed as AP + en, where en is ann-vector of independent identically distributed errors whose common distribution P hasmean zero and finite variance. Here the unknown parameter 0 is the pair (3, P). Letthe roots (Rku) and the index set U be as in case (a) of Example 1.

In this nonparametric setting, Scheffe's simultaneous confidence intervals for thelinear combinations (u'() no longer have overall level 1 - a; nor are they balanced.Nevertheless, under certain assumptions on the regression matrix A (Huber, 1973), theoverall coverage probability of Scheffe's confidence intervals Cn converges to 1 - a,pointwise in 0; and the probability that the component interval C,, contains the linearcombination u'( converges pointwise in 0 to a value independent of u. Both asser-tions about limits remain true if the F distribution with q and n - r degrees of freedomis replaced, in calculating the critical values, by q-1-times the chi-squared distribution

- 4 -

with q degrees of freedom.

1.3. The B-Method

In general, both the exact and asymptotic distributions of the roots (R,l,,) dependon u and on 0; and supuR,,u is not a continuous pivot, even asymptotically. The twomethods described in sections 1.1 and 1.2 then fail to produce a simultaneousconfidence set which is asymptotically balanced and of correct overall coverage proba-bility.

Example 3. Suppose xn is a sample of n independent identically distributed r-vectors whose common distribution P has finite mean vector and finite covariancematrix. The unknown parameter 0 in this case is the distribution P. Suppose theeigenvalues of the covariance matrix are distinct and strictly positive. Define T. (0) tobe the uth eigenvector of the covariance matrix, corresponding to the uth largest eigen-value and expressed as a vector of unit length whose direction is determined by apreset rule. The r ordered orthonormal eigenvectors then define T (0). The range T ofT (0) is the set of all orthonormal systems in r-space.

To construct simultaneous confidence cones for the eigenvectors{Tu(0): 1 s u < r), consider the roots

R u = n(l-Id ', TU(0)I)9 (1.5)

where ct,u is the uth sample eigenvector, standardized to unit length. The asymptoticdistribution of Ru is complicated and depends on both u and the unknown distribution0 (Davis, 1977). The asymptotic distribution of supuRu depends on 0 and is analyti-cally intractable. Neither the pivotal method nor the method of asymptotic pivots canhandle this example.

The method of bootstrapped roots, or B-method, overcomes distibutionaldifficulties such as those in Example 3. Let Hu (- , 0) and Hn(- , 0) be the left-continuous cumulative distribution functions of Rnu and of supuHnu (R,,,u 0) respec-tively. Suppose on is a good consistent estimate of 0. Let Hnu = H (.u, On) and

= Hn(, O,n) be the natural plug-in estimates of the respective cdfs. In Efron's(1979) terminology, Hu and Hn are bootstrap estimates.

Let H-u (t) and H; (t) denote the largest t-th quantiles of H,u and Hn respectively.The B-method sets the critical values of Cn in (1.2) to be

cu = 'Ku IAl (1 - a)]. (1.6)

- 5 -

An intuitive rationale for this choice is presented in section 2. 1. Each critical value cUis obtained in two steps: (a) Find the largest (1 - a)th quantile of EH and call it b; (b)Find the largest bth quantile of H.U*. This is cu. Monte Carlo approximations for H,,uand H, are discussed in section 2.4.

Under general conditions, the B-method generates a simultaneous confidence setwhich, asymptotically, is balanced and has correct overall coverage probability. Exam-ple 3 satisfies these conditions (Beran, 1988). The B-method is, in fact, a logicalextension of the exact pivotal method described earlier. If the roots (Ru) are identi-cally distrbuted pivots and supuRu is a continuous pivot, then the B-method isequivalent to the pivotal method. In particular, applying the B-method to Example 1yields the Scheffe and Tukey simultaneous confidence intervals in the normal linearmodel.

1.4. The B2-MethodAt finite sample sizes, a B-method simultaneous confidence set C, may suffer error

in its overall coverage probability and lack of balance among the marginal coverageprobabilities of its constituent confidence sets (Cju. The B2-method, or method ofdoubly bootstrapped roots, has the important property that it reduces both kinds oferror, at least for moderate to large sample sizes. It is defined as follows. Let

ST,M = Hn I HnU (Rnu)] (1.7)

Apply the B-method to the transformed roots (Snu) rather than to the original roots(R u). The simultaneous confidence set so obtained is called the B2-methodconfidence set for T (9).

More explicitly, let K (,u 0) and Kn( ,0) be the left-continuous cumulative dis-tribution functions of Snu and of supu K,nu (Suu 0) respectively. Let Klr.u = KYu (n*9o)and Kn = Kn ( * On) denote the corresponding bootstrap estimates of these cdf's; and letK%;; and K, denote the quantile functions of these estimates, defined as in section 1.3.The B2-method sets the critical values in (1.2) to be

= HA; [-1 (Kj.u [ K- (1- a) (1.8)

The evaluation of (1.8) is more complicated but otherwise similar to that of (1.6). Anintuitive argument for the critical values (1.8) is given in section 2.2. Monte Carloapproximations for Kn.,U Kn, Hu, Hn are described in section 2.4.

The B2-method has several noteworthy properties. It is widely applicable. Underasymptotic expansion assumptions discussed in section 3, the B2-method reduces theasymptotic order of imbalance in the B-method simultaneous confidence set; and at thesame time it reduces the asymptotic order of error in overall coverage probability.

- 6 -

Moreover, like the B-method, the B2-method is equivalent to the pivotal method in thecircumstances already described at the end of section 1.3. In the special case wherethe index set U has only one element, the B2-method reduces to the double bootstrapconfidence set studied in Beran (1987). In this sense, the B2-method generalizes theprepivoting transformation discussed in that paper.

Example 4. Suppose that the sample xn, the parametric functions (Tu(0)), and theestimate On = (on, Sn) are as in Example 1; but that the roots are not studentized:

R.n.U= IU' (ONn3)I (1.9)

Consider the B-method and B2-method in the case where U is a subspace of dimensionq.

Let P denote the cumulative distribution function of I Z I. where Z has a standardnormal distribution. Evidently,

Hmu( -X) = vP(*/cu) (1.10)

with Yu2=u'(A' A)f1u c2. Let-xq be the cumulative distribution function of V1/2,where V has a chi-squared distribution with q degrees of freedom. Then, by theCauchy-Schwarz inequality,

Hn(- X)=. -1. (111

In this setting, the B-method thus yields an asymptotic approximation to Scheff6s(1953) simultaneous confidence intervals for the linear combinations {u',1), in whichq-times the (1 - a)th quantile of the F-distribution with q and n - r degrees of free-dom is replaced by the (1 - a)th quantile of the chi-squared distribution with qdegrees of freedom. This simultaneous confidence set is not exact but is asymptoti-cally correct, both in level and in balance.

On the other hand, the induced roots (Snu), defined in (1.7), are given here by

Sn.u = Xq [I U" (On - 13) I/ dn,u ] (1.12)with drnu as in Example 1. Consequently, the B2-method in this setting yieldsScheffe's exact simultaneous confidence intervals for the (u'1); the argument is simi-lar to that for part (a) of Example 1. This simultaneous confidence set is balanced andhas correct asymptotic level at every sample size.

- 7 -

2. BASIC THEORY

Discussed in this section are three related topics: intuitive derivations for the B-method and B2-method simultaneous confidence sets; estimates for the imbalance andoverall coverage probability of the B-method simultaneous confidence set; and relevantMonte Carlo approximations. The concepts and notation from section 1 are retained.

2.1. Motivating the B-Method

Consider a simultaneous confidence set Cn of the form (1.2), where the criticalvalues (c.) are constants to be determined. The choice of the (c}) is subject to therequirements

Pe,n[C3 T(O)] =P,n[[Ru<.cuforeveryue U] (2.1)-1-a

and, for every u in U,

Pe,n [Cu 3 Tu(O)] = P0, n I R .u< CU (2.2)

where 1 is unspecified but does not depend on u. Condition (2.1) sets the overall cov-erage probability and condition (2.2) ensures balance. If 0 is known, (2.1) and (2.2)determine the critical values (cu). Indeed, when H,u and H1l are continuous, (2.2) hasthe solution cU = H 1 (1, 0); and then (2.1) implies 13 = H1 (1 - a, 0). Consequently

cu = FHu[H (1l-a,0),0] (2.3)

solves the system of equations (2.1) and (2.2).

Since 0 is actually unknown, it is reasonable to substitute the estimate (n for 0 inthe critical values (2.3). The end result of this argument is the B-method simultaneousconfidence set for T (0),

C.B = (t e T: Ru(xv,tu) H;u[IH (1 - a)] for every u e U) (2.4)which was previously introduced in section 1.3. Let S,u = Hn [ Hnu (R,,,u) ] as in sec-tion 1.4. Another way of writing C.B is

CA,B = (t e T : Sn,u(x,tu) < 1 - a for every u e U). (2.5)

The basic asymptotic theory for Cn.B rests on representation (2.5) and two conver-gences which occur under general conditions: (a) the asymptotic distribution ofHn,u (Ru) is uniform on (0, 1) for every value of u; (b) the asymptotic distribution ofsupuSnu is also uniform on (0, 1). Property (b) ensures that the asymptotic coverageprobability of C(B for T (0) is 1 - a. Property (a) ensures the asymptotic balance of

- 8-

Cl],B For more details and for proofs, see Beran (1988).

2.2. Motivating the B2-Method

In the form (2.5), the B-method simultaneous confidence set refers supu Snu to the(1 - a)th quantile of the uniform distribution on (0, 1). To improve upon this choiceof critical values for CKB, consider more generally a simultaneous confidence set forT (0) of the form

(t e T: Sn(xnttu) . du for every u e U), (2.6)

where the (d,j are to be determined. Repeating the argument for the B-method insection 2.1 yields the critical values

du = Kn[uI (1 -a)b] (2.7)

where Knu and Kn are the bootstrap distributions defined in section 1.4.

The simultaneous confidence set for T(O) which this reasoning generates can bewritten in two other equivalent forms:

CnB2 = te T: Kn[Kn,U(Sn,u(xn,t,))Ls1 - a for every u e U) (2.8)

- (t e T : Rmu (xn, tu) . H [H,1(K'[Ku (1 - a)]] for every ue U).

The second expression in (2.8) coincides with the B2-method simultaneous confidenceset defined in section 1.4.

Since the B2-method is a double iteration of the B-method, the confidence set C. B2inherits the basic asymptotic properties of CnB - asymptotic balance and asymptoticcoverage probability 1 - a - under general conditions. That CnB2 is actually asymp-totically better than CnB can be seen in two ways: through studying special cases suchas Example 4 of section 1.4 and Example 5 of section 2.4; and through the analysis insection 3, which is based on asymptotic expansions.

2.3. Estimating Coverage Probability and Imbalance

As in section 1.1, let Cn be any simultaneous confidence set for T(O) constructedfrom component confidence sets (CJ). The overall coverage probability of Cn andthe uth component coverage probability of C, are

CP(0) = P.n[I 3 T(0)] (2.9)

CPU(0) = P,n[CCnu 3 Tu(0)]respectively. Define the imbalance of Cn to be

IM (0) = I sup. CP. (0) - inf. CPU (0) 1. (2.10)

- 9 -

The confidence set C, is balanced if and only if the imbalance IM (0) vanishes.

Good natural estimates for the coverage probabilities and for the imbalances areCP (0n), CPU (On) and IM (On) respectively. For the B-method simultaneous confidenceset Cn,B, these bootstrap estimates can be rewritten usefully. Define the cdf'sK4(u 0) and Knu as in section 1.4. Let Kn'( ,0) be the right-continuous cdf ofsupU Snu and let Kn'=K'( ,On) be its bootstrap estimate. Evidently, the coverageprobability of CnB is

CPB(O) = Kn'(1-ca,0) (2.11)and the uth component coverage probability and the imbalance of Cn B are

CPB,u(0) = K4U[(1 - a)+, 0] (2.12)IMB (0) = supu CPB,U(0) - infu CPB,U (0)1

Replacing K3u, KIC, in (2.11) and (2.12) with Knu Kn' respectively yields the bootstrapestimates CPB (0n), CPB,U (Os), and IMB (On). Section 3.3 analyzes the rates of conver-

gence of these estimates. The Monte Carlo algorithm which approximates the B2-method simultaneous confidence set also approximates CPB (On), CPB,u(On) andIMB (On); section 2.4 presents the details.

Example 5. Consider the normal r-sample model, in which xn is the union of rindependent subsamples of sizes (nu: 1 s u < r} respectively and n denotes the overallsample size. The uth subsample consists of nu independent identically distributedobservations from the N (t,, a2) distribution. The unknown parameter 0 is(1,... , p.12 ,. . . , ar). Let (A,U9 d2u) denote the usual estimates of (pg, ia2).Suppose that the parametric functions of interest are the means

Tu(0) = 1Ul < u < r (2.13)

and that the corresponding roots are

RU = nu12 I gu (2.14)

For T as in Example 4,

Hn,u (x. 0) = v (x / au)

Hln(x,0) =xrforO x. (2.15)

Consequently, the B-method simultaneous confidence set for p. = (,. . . , A) is

Cn,B = L: n'1/21,u --u I ' dnuT1 [(1 - a)l/r] for 1 < u < r). (2.16)Let 'P denote the cumulative distribution function of IW I when W has a t-distributionwith v degrees of freedom. The marginal level of the uth component confidence

- 10-

interval in (2.16) is ',P1 - I- [(1 - a)1"]. The overall level of C,B is the product

of the r marginal levels. In this example, CPB (0) and IMB (0) do not depend on 0;consequently, their natural estimates are exact.

From (2.14) and (2.15), it follows that the transformed roots (Snu) for the B2-method are

Sn,u = {[In /2 1 u- Lul/nu (2.17)

Thus,

CI!.,B2 = { -nul/2 Iku s dnu%F 1 -a) 1r] for 1. u < r}. (2.18)

The overall level of C,,B2 is 1 - a; the marginal levels are each (1 - a)11'. The B2-method simultaneous confidence set is exact and balanced in this example. Table 1,on the other hand, records the imbalance and overall level of CnB in the two-samplecase (r = 2), when sample sizes are not large. In this situation, the improvementachieved by the B2-method is worthwhile.

- 11 -

Table 1. Overall level, imbalance, and marginal levels of Cn,B and CnB2 in thetwo-sample normal model of Example 5. Nominal overall level is .95.

Confidence Subsample sizes Overall Imbalance Marginal levels for

set (n1, n2) level 1 p.2

(5,5) .830 0 .911 .911(5,10) .864 .037 .911 .948

CnAB (5,20) .877 .051 .911 .962(10,10) .898 0 .948 .948(10,20) .912 .014 .948 .962(20,20) .926 0 .962 .962

Cnl,B2 (nl,n2) .950 0 .975 .975

- 12 -

2.4. Monte Carlo ApproximationsApart from simple special cases, closed fonn expressions are not available for the

bootstrap cumulative distribution functions Hn,U Hn Knu and Kn which are required bythe B-method and B2-method. Analytical approximations, based on asymptotic expan-sions, are occasionally available for some or all of these distribution functions. In gen-eral, Monte Carlo methods provide useful approximations to these bootstrap cdf's andto the estimates of overall coverage probability and of imbalance which were intro-duced in section 2.3.

Underlying the Monte Carlo algorithm are the following representations. Given thesample xn, let x* be a bootstrap sample of size n drawn from the fitted model P n.Let On* = On(xl) denote the estimate of 0 recalculated from the bootstrap sample x*.

Given xn and x*, let x* be a bootstrap sample drawn from the re-fitted model P., n.Write

P-;= Rnu (Xn xO n)

Ru R=nu(xn On) (2.19)

Sn,u =Sn u (xn Xon)Then

Hn,u(x) = POnI['tnou < xXnxH^(x) = * (2.20)Hn(x) = Pd. n I suPu Hnou (Rn,u) < x IXn]

Define

Hn*u (x) = (X, = Po* n[R;u < x Ixn*,x]FHn (x) =Hn (x, On) Po. n[supuIHS(R)uxHx ]u) (2.21)

Since Su = Hn Hnu (Rnu), it follows from (2.21) that

Sn,u = Hn *Hn u(Rn) (2.22)= P:,n [ supuH;u(Rl;u) < HZu (Rx;u) IXn 9X]

Then

K,u = Pd.n[Snu < xIXn

Kn (x) = n I suPu Knu (STi) < XxIn (2.23)

Kn'(X) = n[suPuSnu XXxn]

- 13 -

These representations for Hn,U, Hn, Kl,b, K. show that the B-method is a singlebootstrap procedure while the B2-method is a double bootstrap procedure.

The representations also motivate the following Monte Carlo algorithm for approxi-mating the cdf's H., H., K,u K. required by the B-method and B2-method; and forapproximating the additional cdf K.' required by the coverage probability estimateCPB (On) of section 2.3. The algorithm has five steps:

(a) Let Yi' . y., YM be M bootstrap samples of size n, each drawn from the fittedmodel PO n. These samples are conditionally independent, given the original sample

xI. Set R = R,,U (Yj9 On), the uth root recalculated from the jth bootstrap sample andthe estimate On. The left-continuous empirical cdf of the values (Ruj: 1 . j < M)approximates Hnu for sufficiently large M.

(b) Let

Vj = M- supu[rank(Ru*j)- 1], (2.24)

the rank being calculated over the subscript j. The left-continuous empirical cdf of thevalues (Vj: 1 . j 5 M) approximates Hn for sufficiently large M.

(c) Let = On(Yj*) denote the estimate of 0 recomputed from the jth bootstrapsample. For every j, let yyjN..., Y,, be N further bootstrap samples of size n, eachdrawn from the refitted model P0* n. These samples are conditonally independent,given the original sample xn and the first-round bootstrap samples (yi*: 1. j 5 M).Set Rkjk =Ru (yj, O*)j the uth root recalculated from the (j, k)th second-roundbootstrap sample and the estimate (OJ*. For every u and j, let Zuj be the fraction of thevalues (RU*k: 1 < k < N) which are less than Ru*j. Let

= N1 supu[rank(Ru,)- 1], (2.25)

the rank being calculated over the subscript k. For every u and j, let S *. be the frac-tion of the values {Vjk 1 < k . N) which are less than Zuj. The left-continuousempirical cdf of the values (S* :1 . j s M) approximates K,,u for sufficiently largeM and N.

(d) Repeat step (b) with the {S*: 1. j . M) in place of the (Ru*j:1. j . M).This yields an approximation to Kn for sufficiently large M and N.

(e) For every j, let

S = supuS (2.26)The right-continuous cdf of the values {Sj*: 1 5 j . M) approximates K,' forsufficiently large M and N.

- 14 -

No good theoretical guidelines yet exist for the choice of M and N in this algo-rithm. A practical strategy is to increase M and N until the desired critical values forthe B-method and B2-method simultaneous confidence sets stabilize. Preliminary trialsof the algorithm suggest taking both M and N equal to at least 2000 for 90% simul-taneous confidence intervals.

Example 6. Mardia, Kent, and Bibby (1979) reported test scores for 88 collegestudents, each of whom took two closed-book and three open-book tests. Were theredifferences among the open book tests? We can address this question by constructingsimultaneous confidence intervals for the pairwise differences of the three open-booktest score means, on the supposition that the test scores for each student are a randomvector drawn from a common unknown distribution; and by also constructing simul-taneous confidence intervals for the pairwise ratios of the three open-book test scorestandard deviations.

To better bring out the effects of the B2-method as a refinement of the B-method,we artificially base the analysis on only the first 22 of the 88 test score vectors. Forthis subsample, the sample mean vector and sample covariance matrix of the open-book test scores are

[62.55] [40.74 21.38 61.647 = 161.09 £ = 121.38 34.66 42.23 . (2.27)

59.64J 61.64 42.23 194.34

The pairwise differences of sample means are thus

- l - 42 = 1.46 A1- 13 = 2.91 2 - 23 =1.45 (2.28)and the pairwise ratios of sample standard deviations are

,/ 62 = 1.08 61/ 3 = .458 d2/d3 = .422. (2.29)

Let the (gj± and (asi) denote the population means and standard deviations. Asroots for the desired simultaneous confidence sets, we use

Rnj,j = I (1k. - Aj) - (gi - gj) 1 (2.30)in the case of the differences in means and

R,,,ij = max (rj,, r1j), (2.31)

where

r= (dri / B) / (ai /ai) (2.32)

in the case of the ratios of standard deviations. In both cases, the index u = (i,j)ranges over the set of values U = ((1, 2), (1, 3), (2,3)). Table 2 records the critical

- 15 -

values generated for these two sets of roots (2.30) and (2.31), at nominal level .90 bythe B-method and by the B2-method. Table 3 reports the estimated overall coverageprobability, marginal coverage probabilities, and imbalance of the B-method simultane-ous confidence sets, using the bootstrap estimates of section 2.3.

The estimates in Table 3 indicate that the B-method simultaneous confidence setsfor differences of means and for ratios of standard deviations are nearly balanced,despite the small sample size; however, the overall coverage probability in both casesis wefl below the intended .90. The B2-method accordingly adjusts most criticalvalues upwards, as seen in Table 2. The last digits in Table 2 are not numericallystable, in the sense that increasing the numbers M and N of bootstrap samples beyondM = N = 2000 might cause these last digits to change. However, M = N = 2000 wasthe practical limit on the SUN 3/140 workstation used to perform the calculation forthis example.

For the subsample under discussion, the B2-method at nominal level .90 yields thesimultaneous confidence intervals

(.66, 1.77) (.30,.71) (.30,.60) (2.33)

for the pairwise ratios of standard deviations, taken in the order of (2.29). Thecorresponding B-method intervals are

(.65, 1.79) (.32,.66) (.30, .64) (2.34)

Scores from the third open book test are thus significantly more variable than scoresfrom the other two open book tests. On the other hand, neither method finds interest-ing pairwise differences among the three mean scores.

- 16 -

Table 2. Critical values defining C,,B and CJ,B2 for the two sets ofroots in Example 6. Nominal overall level is .90 in each case.The Monte Carlo approximation is based on 200&2 double bootstrap samples.

Sample Confidence Critical values forsize n set Differences of means Ratios of standard deviations

(1,2) (1,3) (2,3) (1,2) (1,3) (2,3)

22 C^B 2.27 4.27 4.77 1.66 1.44 1.40Cn.B2 2.45 4.86 4.91 1.64 1.55 :1.42

Table 3. Estimated overall coverage probability, imbalance, and marginal coverage probabilities of C4B

for the two sets of roots in Example 6. Nominal overall level is .90 in each case.

The Monte Carlo approximation is based on 20002 double bootstrap samples.

Sample Confidence Overall coverage Imbalance Marginal coverage

size n set for probability probabilities for

(1,2) (1,3) (2,3)

differences of .851 .009 .937 .930 .939means

22ratios of .828 .027 .923 .913 .940

standard deviations

- 17 -

3. ASYMPTOTIC ANALYSIS

The analysis in this section compares the overall and marginal coverage probabili-ties of the B-method and of the B2-method simultaneous confidence sets. Rates ofconvergence are determined for the estimated coverage probabilities and estimatedimbalance of the B-method simultaneous confidence set. The treatment rests on formalasymptotic expansions.

3.1. Coverage Probabilities of the B-Method

Suppose that the estimate On is n12-consistent and that for some non-negativeintegers r and s, the following asymptotic expansions hold, uniformly in x and u andlocally uniformly in 0:

Hnx (x, 0) = u (x) + n-r2 hu (x, 0) + 0 (n-r+ln)Hn(x,0) = Hn(x) + n-,2h(x,0) +O(n-s+l')2) (3.1)

The first term in each expansion does not depend on 0 and is of order 0 (1) in n. Thenon-negative integers r, s thus indicate the order of dependence on 0. Suppose, as istypically true, that the functions on the right side of (3.1) are differentiable in x and 0.Let t = mn (r, s). We will argue below that the overall coverage probability andimbalance of C,B normally satisfy

CPB(O) = 1 - a +O(n-t+l)/2)IMB(O) = O(n<t+lY2) (3.2)

pointwise in 0.

This is the leading case, which arises in constructing simultaneous one-sidedconfidence intervals. Other cases arise in which some of the higher order terms inexpansions (3.1) may vanish. The analysis of these other possibilities parallels thetreatment given here for the leading case.

From (3.1)

Hn [ HX, (x,0)X0] = Hn - Hnu (x) + n-t/2 iu (x, 0) +O (n<t+l)/2) (3.3)

uniformly in x and u and locally uniformly in 0. Hence

sn,U = Hn HU(Ru(3.4)= H (R,, 0 ] + Op (n-t+1Y2)

Since the jumps in Hnu ( *, 0) are at most of order 0 (n(r+l)/2),

Pe, n I Hnu (Rn.uq 3) x I = U (x) +O (n-r+l)/2) (3.5)

where U is the uniform (0, 1) cdf. Thus,

- 18 -

Po,n I Hn IHnu (Pn,ug 0).0.< x I = HFV (x,0) + 0 (n7(r+')12) (3.6)

Consequently, in regular cases K1,su, the left-continuous cdf of S,,,, has an expansionof the form

K'u (X, 0) = f14l (x, 0) + n-<1+Y2ku (x, 0) + 0 (nf(t+2Y2) (3.7)

uniformly in x and u.

Consider next Kn', the right-continuous cdf of supu Sn,. In regular cases, Kn'differs by a term of order 0 (n-t+lY2) from the right-continuous cdf of

Hn [ sUPu Hn,u (Rmug 0), 0 ], because of (3.4). Hence,

K'n (x, 0) = U (x) + n-t+lY2v (x, 0) + 0 (n-(t+l)a2) (3.8)

uniformly in x.

On the other hand, it follows from (3.7) and the definition of Kn in section 1.4 that

Kn differs by a term of order 0 (n-t+lY2) from the left-continuous cdf of

H;'l (supU Sn,u 0). In regular cases, because of (3.8),

Kn (x, 0) = Hn (x, 0) + n<(t+l)12 k (x, 0) + 0 (n<t+2)/2) (3.9)uniformly in x.

From. (3.8), the overall coverage probability of CnB for T (0) satisfies the first lineof (3.2). From (3.7), the marginal coverage probability of the uth componentconfidence set in CE,,B iS

CPB,u(0) = ES-1 (1 - a,0) + n7(t+l)a2ku ((1 - a)+, 0) + O(n-(t+2Y2) (3.10)

uniformly in u. Hence the imbalance IMB (0) satisfies the second relation in (3.2).

3.2. Coverage Probabilities of the B2-Method

By continuing the argument begun in the previous subsection, we will show thatthe overall coverage probability and imbalance of Cn, B2 normally satisfy

CPB2 (0) = 1 - a + 0 (n-t+2)a2)IMB2(0) = 0 (n(t+2)2) (3.11)

pointwise in 0. Comparing (3.11) with (3.2) establishes the principal conclusion of thepaper: under regularity conditions, the B2-method reduces the asymptotic order ofimbalance in the B-method simultaneous confidence set; and at the same time itreduces the asymptotic order of error in overall coverage probability.

Define Tnu = Kn [ Knu (S,u)]. In view of (2.8), the B2-method simultaneousconfidence set can be written as

- 19 -

CB2 = (t e T: sup.T...(x., 1 - a). (3.12)

From (3.7) and (3.9),

K.[KI,.(x,0),0] = U (x) + n-(t+lY2ju(X,0) + 0 (n-(t+2)/2) (3.13)

uniformly in x and u and locally unifonnly in 0. Hence

Tnu = Kn [ K,u (Snug 0), 0 ] + Op (n-(t+2Y2). (3.14)

Moreover, by analogy with (3.6),

PO,n [Kn Krsu(Snul)0 X] = K;1 (x,0) + O(n-(t+2Y2), (3.15)the discontinuities in Knu being at most of order 0 (n(t+2Y2), assuming k, is continu-ous in x. Consequently, in regular cases M,u the left-continuous cdf of Tnu has anexpansion of the form

Mn (x, 0) = K11 (x, 0) + n<t+2Y2 mu(x,0) + 0 (n-t+3)2) (3.16)

unifornly in x and u.

Consider next Ml', the right-continuous cdf of supuTnu. In regular cases, Mn'differs by a term of order 0 (n(t+2Y2) from the right-continuous cdf of

Kn [ supu Knu (Sn,u, 0), 0], because of (3.14). Hence,

Mn' (x, 0) = U (x) + n<(t+2Y2w (x, 0) + O (n<t+3)/2) (3.17)

uniformly in x.

On the other hand, it follows from (3.16) that Mn, the left-continuous cdf of

supuMnu (TT,U, 0), differs by a term of order Op (n7('+2Y2) from the left-continuous cdfof Kn- (supu Tnu,, 0). In regular cases, because of (3.17),

Mn (x, 0) = Kn (x, 0) + ntt+2)/2m (x, 0) + 0 (n-t+3Y2) (3.18)

uniformly in x.

The assertions in (3.11) about the overall coverage probability and imbalance of

C,,B2 follow from (3.17) and (3.16) respectively.

3.3. Convergence of Coverage Probability Estimates

How good are the bootstrap estimates CPB (On) and IMB (0n) of the overall cover-age probability CPB (0) and imbalance IMB (0) of the B-method simultaneousconfidence set? It follows from (2.11) and (3.8) that

CPB (On) = CPB (0) + Op(n t+2Y). (3.19)The imbalance IMB (0) converges to zero asymptotically. From (2.12) and (3.7),

- 20 -

IMB (On) = IMB (O)+ Op(n ). (3.20)

By contrast, consider the simpler asymptotic. estimates of overall coverage proba-bility and of imbalance: CPB = 1 - a and IMB = 0. It is immediate from (3.2) that

CPB = CPB (8) + 0 (1f(t+1Y2) (3.21)

and

IMB -IMB (0) + O (n<t+ )/2). (3.22)

Thus, the bootstrap estimates dominate the simple asymptotic estimates of CPB (8) andIMB (8) for all sufficiently large n.

Convergence of CPB,U (On) to the uth component coverage probability CPBu (0) is amore complicated matter. Because of (3.10) and (3.1),

CPB,u (on) = CPB,u(8) + Op(n-s+lY2) + O, (n-t+2Y,2). (3.23)

The rate of convergence in (3.23) can be slower than that in (3.19).

- 21 -

4. COMPLEMENTS

This final section teats several further questions concerning the B2-method: thedistinction between level and coverage probability; computational strategies for B2-method critical values when the index set U is infinite; and the asymptotic propertiesof higher-order iterates of the B-method.

4.1. Level and Imbalance

By the analysis of section 3.2, confidence set CnB2 has the property that its overallcoverage probability converges pointwise to 1 - a,

CPB2(9) = 1-a + 0 (n-t+2)a2) for every 09 (4.1)

and that its imbalance converges pointwise to zero,

IMB2 (9) = 0 (n(t+2)/2) for every 9. (4.2)

Here t is a non-negative integer, defined in section 3.1, which depends upon the'parametric model and the roots (R.,.). These two convergences are not very strongbecause the remainder terms on the right-side of (4.1) and (4.2) could remain large forsome value of 9 no matter how great the sample size n might be.

More satisfactory would be uniform rates of convergence for overall coverage pro-bability and imbalance:

SUP0 I CPB2 (9) - (1 - a) I = 0 (n-t+2Y2)

Sup@ IMB2 (9) = 0 (h-(t+2Y2)* (4.3)

The asymptotic expansions in sections 3.1 and 3.2 typically converge uniformly in 9,as long as 9 is restricted to a compact set. Consequently (4.3) holds if the parameterspace 0 lies within a compact set. In particular, the first line of (4.3) implies a rate ofconvergence for the overall level of C,,BI2:

info CPB2 (9) = 1 - a + 0 (nt1+2)a2). (4.4)

Unfortunately, when 9 is infinite dimensional, the parameter space e usually does notlie within a compact and (4.3) breaks down.

The notion of controlling the overall level and the marginal levels of a simultane-ous confidence set thus seems too strong. More accessible is the goal of controllingaverage coverage probabilities. We conjecture that, under regularity conditions,

ICPB2(9)7 (dO) = 1 - a + o(nt+2)/2), (4.5)where x is any member of some large class of prior distributions supported on 19. Theaverage coverage probability (4.5) makes sense if we think of nature selecting the true

- 22 -

o from a prior distribution i. For an interesting technical discussion of average cover-age probabilities, see Woodroofe (1986).

4.2. Critical Values for Infinite U

The Monte Carlo algorithm for B2-method critical values (see section 2.4) requirescalculation of four suprema over U. Doing this exactly may not be possible when thenumber of elements in U is infinite. A natural approximation strategy is to replacesupremum over U by maximum over a well-chosen finite subset of U, which may bedeterministic or random.

Standard numerical algorithms for maximization search over a deterministic subsetof U. Altematively, the search subset might consist of a random sample drawn from adistribution on U which has full support (that is, the distribution gives positive proba-bility to every non-empty open subset of U). More complex random search schemesalso exist. When the dimension of U is not low, a random search usually needs fewerpoints to approximate a supremum adequately than does a deterministic search. Forfurther analysis of this property, in connection with the B-method, see Beran (1988).

4.3. The BJ-MethodTwo iterations of the B-method gave the B2-method. The process may be contin-

ued: j iterations of the B-method define the Bi-method. We illustrate this by describ-ing the B3-method explicitly. As in section 3.2, let

T4u = K.[Ku(Snu) (4.6)

where Snu is defined by (1.7). The transformed roots (Tnu) are the outcome of theB2-method. Applying the B-method to the roots (Tnu) yields the B3-method simul-taneous confidence set.

In other words, let L,,( , 0) and Ln( ,0) be the left-continuous cdf's of Tn.. andof supu Lnu (T4U, 0) respectively. Let L4U = Lu ( on) and Ln = Ln ( ' .n) denote thecorresponding bootstrap estimates of these cdfs. The B3-method sets the critical valuesin simultaneous confidence set (1.2) to be

Cu= A l [lAl [LAl (l-a)])])]. (4.7)

This expression may be compared with (1.8) for the B2-method and with (1.6) for theB-method.

Under regularity conditions, the overall coverage probability and imbalance ofCnJ,BJ satisfy

CPBj (0) = 1 - a + 0 (n<(t+ji2)IMBJ (0) = 0 (n't+#)/2) (4.8)

- 23 -

pointwise in 0. The argument is an iteration of that in section 3.2. Several features ofthis result are noteworthy:

(a) For all sufficiently large n, the error in coverage probability and the imbalanceof C.,BjH1 will be smaller than their counterparts for Cn Bj. However, for fixed samplesize n - the situation usually encountered in practice - there is no assurance that theerror in coverage probability and imbalance of CBj will decrease as j increases.

(b) Constructing CBj, whether exactly or by some combination of Monte Carlo andanalytical approximations, is rarely feasible, at present, for j 2 3.

(c) The Bh-method falls into a very general class of iterated bootstrap methodsrecently identified by Hall and Martin (1988). A common feature of these bootstrapmethods is that they seek to solve a system of equations analogous to the system (2.1)and (2.2).

- 24 -

REFERENCES

BERAN, R. (1987). "Prepivoting to reduce level error of confidence sets," Biome-trika, 74, 457-468.

BERAN, R. (1988). "Balanced simultaneous confidence sets," Journal of the Ameri-can Statistical Association, 83, 679-697.

DAVIS, A.W. (1977). "Asymptotic theory for principal component analysis: nonnor-mal case," Australian Journal of Statistics, 19, 206-212.

EFRON, B. (1979). "Bootstrap methods: another look at the jackknife," Annals ofMathematical Statistics, 7, 1-26

HALL, P. and MARTIN, M.A. (1988). "On bootstrap resampling and iteration,"Biometrika, 75, 661-671.

HUBER, P.J. (1973). "Robust regression: asymptotics, conjectures, and MonteCarlo," Annals of Statistics, 1, 799-821.

MARDIA, K.V., KENT, J.T., and BIBBY, J.M. (1979). Multivariate Analysis, NewYork: Academic.

MILLER, R. (1966). Simultaneous Statistical Inference, New York: McGraw-Hill.

SCHEFFE, H. (1953). "A method for judging all contasts in the analysis of vari-ance," Biometrika, 40, 87-104.

TUKEY, J.W. (1953). "The problem of multiple comparisons," Unpublished preprint.

WOODROOFE, M. (1986). "Very weak expansions for sequential confidence lev-els," Annals of Statistics, 14, 1049-1067.

TECHNICAL REPORTSStatistics Department

University of California, Berkeley

1. BREIMAN, L. and FREEDMAN, D. (Nov. 1981, revised Feb. 1982). How many variables should be entered in aregression equation? Jour. Amer. Statist. Assoc., March 1983, 78, No. 381, 131-136.

2. BRILLINGER, D. R. (Jan. 1982). Some contasting exnamples of the time and frequency domain approaches to time seriesanalysis. Time Series Methods in Hydroscmences, (A. H. El-Shaarawi and S. R. Esterby, eds.) Elsevier ScientificPublishing Co., Amsterdam, 1982, pp. 1-15.

3. DOKSUM, K. A. (Jan. 1982). On the performance of estimates in portional hazard and log-linear models. SurvivalAnalysis, (John Crowley and Richard A. Johnson, eds.) IMS Lecture Notes - Monograph Series, (Shanti S. Gupta, seriesed.) 1982, 74-84.

4. BICKEL, P. J. and BRE[MAN, L. (Feb. 1982). Sums of functions of nearest neighbor distances, moment bounds, limittheorems and a goodness of fit test. Ann. Prob., Feb. 1982, 11. No. 1, 185-214.

5. BRILLINGER, D. R. and TUKEY, J. W. (March 1982). Spectrum estimation and system identification relying on aFourier transform. The Collected Works of J. W. Tukey, vol. 2, Wadsworth, 1985, 1001-1141.

6. BERAN, R. (May 1982). Jackknife approximation to bootstrap estimates. Ann. Statist., March 1984, 12 No. 1, 101-118.

7. BICKEL, P. J. and FREEDMAN, D. A. (June 1982). Bootstrapping regression models with many parameters.Lehmann Festschrift, (P. J. Bickel, K. Doksum and J. L. Hodges, Jr., eds.) Wadsworth Press, Belmont, 1983, 2848.

8. BICKEL, P. J. and COLLINS, J. (March 1982). Minimizing Fisher information over mixtures of distributions. Sankhyi,1983, 45, Series A, Pt. 1, 1-19.

9. BREIMAN, L. and FRIEDMAN, J. (July 1982). Estimating optimal transformations for multiple regression and correlation.

10. FREEDMAN, D. A. and PETERS, S. (July 1982, revised Aug. 1983). Bootstrapping a regression equation: someempirical results. JASA, 1984, 79, 97-106.

11. EATON, M. L. and FREEDMAN, D. A. (Sept. 1982). A remark on adjusting for covariates in multiple regression.

12. BICKEL, P. J. (April 1982). Minimax estimation of the mean of a mean of a nonnal distribution subjct to doing wellat a point. Recent Advances in Statistics, Academic Press, 1983.

14. FREEDMAN, D. A., ROTHENBERG, T. and SUTCH, R. (Oct. 1982). A review of a residential energy end use model.

15. BRILLINGER, D. and PREISLER, H. (Nov. 1982). Maximum likelihood estimation in a latent variable problem. Studiesin Econometrics, Time Series, and Multivariate Statistics, (eds. S. Karlin, T. Amemiya, L. A. Goodman). AcademicPress, New York, 1983, pp. 31-65.

16. BICKEL, P. J. (Nov. 1982). Robust regression based on infinitesimal neighborhoods. Ann. Statist., Dec. 1984, 12,1349-1368.

17. DRAPER, D. C. (Feb. 1983). Rank-based robust analysis of linear models. I. Exposition and review. Statistical Science,1988, Vol.3 No. 2 239-271.

18. DRAPER, D. C. (Feb 1983). Rank-based robust inference in regression models with several observations per cell.

19. FREEDMAN, D. A. and FIENBERG, S. (Feb. 1983, revised April 1983). Statistics and the scientific method, Commentson and reactions to Freedman, A rejoinder to Fienberg's comments. Springer New York 1985 Cohort Analysis in SocialResearch, (W. M. Mason and S. E. Fienberg, eds.).

20. FREEDMAN, D. A. and PETERS, S. C. (March 1983, revised Jan. 1984). Using the bootstrap to evaluate forecastingequations. J. of Forecasting. 1985, Vol. 4, 251-262.

21. FREEDMAN, D. A. and PETERS, S. C. (March 1983, revised Aug. 1983). Bootstrapping an econometic model: someempirical results. JBES, 1985, 2, 150-158.

22. FREEDMAN, D. A. (March 1983). Structural-equation models: a case study.

23. DAGGETT, R. S. and FREEDMAN, D. (April 1983, revised Sept. 1983). Econometrics and the law: a case study in theproof of antitrust damages. Proc. of the Berkeley Conference, in honor of Jerzy Neyman and Jack Kiefer. Vol I pp.123-172. (L. Le Cam, R. Olshen eds.) Wadsworh, 1985.

- 2 -

24. DOKSUM, K. and YANDELL, B. (April 1983). Tests for exponentiality. Handbook of Statistics, (P. R. Krishnaiah andP. K. Sen, eds.) 4, 1984, 579-611.

25. FREEDMAN, D. A. (May 1983). Comments on a paper by Markus.

26. FREEDMAN, D. (Oct. 1983, revised March 1984). On bootstrapping two-stage least-squares estimates in stationary linearmodels. Anm. Statist., 1984, 12, 827-842.

27. DOKSUM, K. A. (Dec. 1983). An extension of partial likelihood methods for proportional hazard models to -generaltransfonnation models. Ann. Statist., 1987, 15, 325-345.

28. BICKEL, P. J., GOETZE, F. and VAN ZWET, W. R. (Jan. 1984). A simple analysis of third order efficiency of estimateProc. of the Neyman-Kiefer Conference, (L. Le Cam, ed.) Wadsworth, 1985.

29. BICKEL, P. J. and FREEDMAN, D. A. Asymptotic normality and the bootstrap in stratified sampling. Ann. Statist.12 470-482.

30. FREEDMAN, D. A. (Jan. 1984). The mean vs. the median: a case study in 4-R Act litigation. JBES. 1985 Vol 3pp. 1-13.

31. STONE, C. J. (Feb. 1984). An asymptotically optimal window selection rule for kemel density estimates. Ann. Statist.,Dec. 1984, 12, 1285-1297.

32. BREIMAN, L. (May 1984). Nail finders, edifices, and Oz.

33. STONE, C. J. (Oct. 1984). Additive regression and other nonparametric models. Ann. Statist., 1985, 13, 689-705.

34. STONE, C. J. (June 1984). An asymptotically optimal histogram selection rule. Proc. of the Berkeley Conf. in Honor of-Jerzy Neyman and Jack Kiefer (L Le Cam and R. A. Olshen, eds.), 11 513-520.

35. FREEDMAN, D. A. and NAVIDL W. C. (Sept. 1984, revised Jan. 1985). Regression models for adjusting the 1980Census. Statistical Science. Feb 1986, Vol. 1, No. 1, 3-39.

36. FREEDMAN, D. A. (Sept. 1984, revised Nov. 1984). De Finetti's theorem in continuous time.

37. DIACONIS, P. and FREEDMAN, D. (Oct. 1984). An elementary proof of Stirling's formula. Amer. Math Monthly. Feb1986, Vol. 93, No. 2, 123-125.

38. LE CAM, L. (Nov. 1984). Sur l'approximation de familles de mesures par des familles Gaussiennes. Ann. Inst.Henri Poincare, 1985, 21, 225-287.

39. DIACONIS, P. and FREEDMAN, D. A. (Nov. 1984). A note on weak star uniformities.

40. BREIMAN, L. and IHAKA, R. (Dec. 1984). Nonlinear discriminant analysis via SCALING and ACE.

41. STONE, C. J. (Jan. 1985). The dimensionality reduction principle for generalized additive models.

42. LE CAM, L. (Jan. 1985). On the nornal approximation for sums of independent variables.

43. BICKEL, P. J. and YAHAV, J. A. (1985). On estimating the number of unseen species: how many executions werethere?

44. BRILLINGER, D. R. (1985). The natural variability of vital rates and associated statistics. Biometrics, to appear.

45. BRILLINGER, D. R. (1985). Fourier inference: some methods for the analysis of array and nonGaussian series data.Water Resources Bulletin, 1985, 21, 743-756.

46. BREIMAN, L. and STONE, C. J. (1985). Broad spectrum estimates and confidence intervals for tail quantiles.

47. DABROWSKA, D. M. and DOKSUM, K. A. (1985, revised March 1987). Partial likelihood in transformation modelswith censored data. Scandinavian J. Statist., 1988, 15, 1-23.

48. HAYCOCK, K. A. and BRILLINGER, D. R. (November 1985). LIBDRB: A subroutine library for elementary timeseries analysis.

49. BRILLINGER, D. R. (October 1985). Fitting cosines: some procedures and some physical examples. Joshi Festschrift,1986. D. Reidel.

50. BRILLINGER, D. R. (November 1985). What do seismology and neurophysiology have in common? - Statistics!Comptes Rendus Math. B Acad. Sci. Canada. January, 1986.

51. COX, D. D. and O'SULLIVAN, F. (October 1985). Analysis of penalized likelihood-type estimators with application togeneralized smoothing in Sobolev Spaces.

- 3 -

52. O'SULLIVAN, F. (November 1985). A practical perspective on ill-posed inverse problems: A review with somenew developments. To appear in Journal of Statistical Science.

53. LE CAM, L. and YANG, G. L. (November 1985, revised March 1987). On the preservation of local asymptotic normalityunder infornation loss.

54. BLACKWEIA D. (November 1985). Approximate nornality of large products.

55. FREEDMAN, D. A. (June 1987). As others see us: A case study in path analysis. Joumal of EducationalStatistics. 12, 101-128.

56. LE CAM, L. and YANG, G. L. (January 1986). Replaced by No. 68.

57. LE CAM, L. (Febuary 1986). On the Bernstein - von Mises theorem.

58. O'SULLIVAN, F. (January 1986). Estimation of Densities and Hazards by the Method of Penalized likelihood.

59. AL1DOUS, D. and DIACONIS, P. (February 1986). Strong Uniform Times and Finite Random Walks.

60. ALDOUS, D. (March 1986). On the Markov Chain simulation Method for Unifonn Combinatorial Distributions andSimulated Annealing.

61. CHENG, C-S. (April 1986). An Optimization Problem with Applications to Optimal Design Theory.

62. CHENG, C-S., MAJUMDAR, D., STUFKEN, J. & TURE, T. E. (May 1986, revised Jan 1987). Optimal step typedesign for comparing test treatments with a control.

63. CHENG, C-S. (May 1986, revised Jan. 1987). An Application of the Kiefer-Wolfowitz Equivalence Theorem.

64. O'SULLIVAN, F. (May 1986). Nonparametric Estimation in the Cox Proportional Hazards Model.

65. ALDOUS, D. (JUNE 1986). Finite-Time Implications of Relaxation Times for Stochastically Monotone Processes.

66. PiTMAN, J. (JULY 1986, revised November 1986). Stationary Excursions.

67. DABROWSKA, D. and DOKSUM, K. (July 1986, revised November 1986). Estimates and confidence intervals formedian and mean life in the proportional hazard model with censored data. Biometrika, 1987, 74, 799-808.

68. LE CAM, L. and YANG, G.L. (July 1986). Distinguished Statistics, Loss of information and a theorem of Robert B.Davies (Fourth edition).

69. STONE, C.J. (July 1986). Asymptotic properties of logspline density estimation.

71. BICKEL, PJ. and YAHAV, J.A. (July 1986). Richardson Extrapolation and the Bootstrap.

72. LEHMANN, E.L. (July 1986). Statistics - an overview.

73. STONE, C.J. (August 1986). A nonparametric framework for statistical modelling.

74. BIANE, PH. and YOR, M. (August 1986). A relation between Levy's stochastic area formula, Legendre polynomial,and some continued fractions of Gauss.

75. LEHMANN, E.L. (August 1986, revised July 1987). Comparing Location Experiments.

76. O'SULLIVAN, F. (September 1986). Relative risk estimation.

77. O'SULLIVAN, F. (September 1986). Deconvolution of episodic hormone data.

78. PITMAN, J. & YOR, M. (September 1987). Further asymptotic laws of planar Brownian motion.

79. FREEDMAN, D.A. & ZEISEL, H. (November 1986). From mouse to man: The quantitative assessment of cancer risks.To appear in Statistical Science.

80. BRILLINGER, D.R. (October 1986). Maximum likelihood analysis of spike trains of interacting nerve cells.

81. DABROWSKA, D.M. (November 1986). Nonparametric regression with censored survival time data.

82. DOKSUM, K.J. and LO, A.Y. (Nov 1986, revised Aug 1988). Consistent and robust Bayes Procedures forLocaton based on Partial Information.

83. DABROWSKA, D.M., DOKSUM, KA. and MIURA, R. (November 1986). Rank estimates in a class of semiparametrictwo-sample models.

- 4 -

84. BRILLINGER, D. (December 1986). Some statistical methods for random process data from seismology andneurophysiology.

85. DIACONIS, P. and FREEDMAN, D. (December 1986). A dozen de Finetti-style results in search of a theory.Ann. Inst. Heni Poincare, 1987, 23, 397-423.

86. DABROWSKA, D.M. (January 1987). Uniform consistency of nearest neighbour and kemel conditional Kaplan- Meier estimates.

87. FREEDMAN, DA., NAVIDI, W. and PETERS, S.C. (February 1987). On the impact of variable selection infitting regression equations.

88. ALDOUS, D. (February 1987, revised April 1987). Hashing with linear probing, under non-uniform probabilities.

89. DABROWSKA, D.M. and DOKSUM, KA. (March 1987, revised January 1988). Estimating and testing in a twosample generalized odds rate model. J. Amer. Statist. Assoc., 1988, 83, 744-749.

90. DABROWSKA, D.M. (March 1987). Rank tests for matched pair experiments with censored data.

91. DIACONIS, P and FREEDMAN, D.A. (April 1988). Conditional limit theorems for exponential families and finiteversions of de Finetti's theorem. To appear in the Joumal of Applied Probability.

92. DABROWSKA, D.M. (April 1987, revised September 1987). Kaplan-Meier estimate on the plane.

92a. ALDOUS, D. (April 1987). The Harmonic mean formula for probabilities of Unions: Applications to sparse randomgraphs.

93. DABROWSKA, D.M. (June 1987, revised Feb 1988). Nonparametric quantile regression with censored data.

94. DONOHO, D.L. & STARK, P.3. (June 1987). Uncertainty principles and signal recovery.

95. CANCELLED

96. BRILLINGER, D.R. (June 1987). Some examples of the statistical analysis of seismological data. To appear inProceedings, Centennial Anniversary Symposium, Seismographic Stations, University of Califomia, Berkeley.

97. FREEDMAN, DA. and NAVIDI, W. (June 1987). On the multi-stage model for carcinogenesis. To appear inEnvironmental Health Perspectives.

98. O'SULLIVAN, F. and WONG, T. (June 1987). Determining a function diffusion coefficient in the heat equation.99. O'SULLIVAN, F. (June 1987). Constrained non-linear regularization with application to some system identification

problems.

100. LE CAM, L. (July 1987, revised Nov 1987). On the standard asymptotic confidence ellipsoids of Wald.

101. DONOHO, D.L. and LIU, R.C. (July 1987). Pathologies of some minimum distance estimators. Annals ofStatistics, June, 1988.

102. BRILLINGER, D.R., DOWNING, K.H. and GLAESER, R.M. (July 1987). Some statistical aspects of low-doseelectron imaging of crystals.

103. LE CAM, L. (August 1987). Harald Cramer and sums of independent random variables.

104. DONOHO, A.W., DONOHO, D.L. and GASKO, M. (August 1987). Macspin: Dynamic graphics on a desktopcomputer. IEEE Computer Graphics and applications, June, 1988.

105. DONOHO, D.L. and LIU, R.C. (August 1987). On minimax estimation of linear functionals.

106. DABROWSKA, D.M. (August 1987). Kaplan-Meier estimate on the plane: weak convergence, LIL and the bootstrap.

107. CHENG, C-S. (Aug 1987, revised Oct 1988). Some orthogonal main-effect plans for asymmetrical factorials.

108. CHENG, C-S. and JACROUX, M. (August 1987). On the construction of trend-free run orders of two-level factorialdesigns.

109. KLASS, M.J. (August 1987). Maximizing E max Sk/ ES': A prophet inequality for sums of I.I.D. mean zero variates.

110. DONOHO, D.L. and LIU, R.C. (August 1987). The "automatic" robustness of minimum distance functionals.Annals of Statistics, June, 1988.

111. BICKEL P.J. and GHOSH, J.K. (August 1987, revised June 1988). A decomposition for the likelihood ratio statisticand the Bartlett correction- a Bayesian argument.

- 5 -

112. BURflZY, K., PITMAN, J.W. and YOR, M. (September 1987). Some asymptotic laws for crossings and excursions.

113. ADHIKART, A. and PrTMAN, L. (September 1987). The shortest planar arc of width 1.

114. R1TOV, Y. (September 1987). Estimation in a linear regression model with censored data.

115. BICKEL, PJ. and RITOV, Y. (Sept. 1987, revised Aug 1988). Large sample theory of estimation in biased samplingregression models I.

116. RITOV, Y. and BICKEL, P.J. (Sept.1987, revised Aug. 1988). Achieving information bounds in non andsemiparametric models.

117. R1ITOV, Y. (October 1987). On the convergence of a maximal correlation algorithm with alternating projections.

118. ALDOUS, D.J. (October 1987). Meeting times for independent Markov chains.

119. HESSE, C.H. (October 1987). An asymptotic expansion for the mean of the passage-time distribution of integratedBrownian Motion.

120. DONOHO, D. and LIU, R. (Oct. 1987, revised Mar. 1988, Oct. 1988). Geometwizing rates of convergence, H.

121. BRILLINGER, D.R. (October 1987). Estimating the chances of large earthquakes by radiocarbon dating and statisticalmodelling. Statistics a Guide to the Unknown, pp. 249-260 (Eds. JIM. Tanur et al.) Wadsworth, Pacific Grove.

122. ALDOUS, D., FLANNERY, B. and PALACIOS, J.L. (November 1987). Two applications of um processes: The fringeanalysis of search trees and the simulation of quasi-stationary distibutions of Markov chains.

123. DONOHO, D.L., MACGIBBON, B. and LIU, R.C. (Nov.1987, revised July 1988). Minimax risk for hyperrectangles.

124. ALDOUS, D. (November 1987). Stopping times and tightness H.

125. HESSE, C.H. (November 1987). The present state of a stochastic model for sedimentation.

126. DALANG, R.C. (December 1987, revised June 1988). Optimal stopping of two-parameter processes onnonstandard probability spaces.

127. Same as No. 133.

128. DONOHO, D. and GASKO, M. (December 1987). Multivariate generalizations of the median and trnmmed mean H.

129. SMITH, D.L. (December 1987). Exponential bounds in Vapnik-tervonenkis classes of index 1.

130. STONE, C.J. (Nov.1987, revised Sept. 1988). Uniform error bounds involving logspline models.

131. Same as No. 140

132. HESSE, C.H. (Dec. 1987, revised June 1989). A Bahadur - Type representation for empirical quantiles of a largeclass of stationary, possibly infinite - variance, linear processes

133. DONOHO, D.L. and GASKO, M. (December 1987). Multivariate generalizations of the median and timmed mean, I.

134. CANCELLED

135. FREEDMAN, D.A and NAVIDI, W. (December 1987). On the risk of lung cancer for ex-smokers.

136. LE CAM, L. (January 1988). On some stochastic models of the effects of radiation on cell survival.

137. DIACONIS, P. and FREEDMAN, DA. (April 1988). On the uniform consistency of Bayes estimates for multinomialprobabilities.

137a. DONOHO, D.L. and LIU, R.C. (1987). Geometrizing rates of convergence, I.

138. DONOHO, D.L. and LIU, R.C. (January 1988). Geometrizing rates of convergence, m.

139. BERAN, R. (January 1988). Refining simultaneous confidence sets.

140. HESSE, C.H. (December 1987). Numerical and statistical aspects of neural networks.

141. BRILUNGER, D.R. (Jan. 1988). Two reports on trend analysis: a) An elemnentary trend analysis of Rio negro levels atManaus, 1903-1985. b) Consistent detection of a monotonic trend superposed on a stationary time series.

142. DONOHO, D.L. (Jan. 1985, revised Jan. 1988). One-sided inference about functionals of a density.

- 6 -

143. DALANG, R.C. (Feb. 1988, revised Nov. 1988). Randomization in the two-armed bandit problem.144. DABROWSKA, D.M., DOKSUM, KA. and SONG, LK. (February. 1988). Graphical comparisons of cumulative hazards

for two populations.145. ALDOUS, D.J. (February 1988). Lower bounds for covering times for reversible Markov Chains and random walks on

graphs.146. BICKEL, P.J. and RITOV, Y. (Feb.1988, revised August 1988). Estimating integrated squared density derivatives.

147. STARK, P.B. (March 1988). Strict bounds and applications.

148. DONOHO, D.L. and STARK, P.B. (March 1988). Rearrangements and smootiing.

149. NOLAN, D. (March 1988). Asymptotics for a multivariate location estimator.

150. SEILLIER, F. (March 1988). Sequential probability forecasts and the probability integral transform.

151. NOLAN, D. (Mar. 1988, revised May 1989). Asymptotics for multivariate trimning.

152. DIACONIS, P. and FREEDMAN, DA. (April 1988). On a theorem of Kuchler and Lauritzen.

153. DIACONIS, P. and FREEDMAN, DA. (April 1988). On the problem of types.

154. DOKSUM, KA. (May 1988). On the correspondence between models in binary regression analysis and survival analysis.

155. LEHMANN, E.L. (May 1988). Jerzy Neyman, 1894-1981.

156. ALDOUS, D.J. (May 1988). Stein's method in a two-dimensional coverage problem.

157. FAN, J. (June 1988). On the optimal rates of convergence for nonparametric deconvolution problem.

158. DABROWSKA, D. (June 1988). Signed-rank tests for censored matched pairs.

159. BERAN, R.J. and MILLAR, P.W. (June 1988). Multivariate symmetry models.

160. BERAN, R.J. and MILLAR, P.W. (June 1988). Tests of fit for logistic models.

161. BREIMAN, L. and PETERS, S. (June 1988). Comparing automatic bivariate smoothers (A public service enterprise).

162. FAN, J. (June 1988). Optimal global rates of convergence for nonparametric deconvolution problem.

163. DIACONIS, P. and FREEDMAN, DA. (June 1988). A singular measure which is locally uniform. (Revised byTech Report No. 180).

164. BICKEL, P.J. and KREEGER, A.M. (July 1988). Confidence bands for a distribution function using the bootstrap.

165. HESSE, C.H. (July 1988). New methods in the analysis of economic time series I.

166. FAN, JLANQING (July 1988). Nonparametric estimation of quadratic functionals in Gaussian white noise.

167. BREIMAN, L., STONE, C.J. and KOOPERBERG, C. (August 1988). Confidence bounds for extreme quantiles.

168. LE CAM, L. (August 1988). Maximum likelihood an introduction.

169. BREIMAN, L. (Aug.1988, revised Feb. 1989). Submodel selection and evaluation in regression I. The X-fixed caseand little bootstrap.

170. LE CAM, L. (September 1988). On the Prokhorov distance between the empirical process and the associated Gaussianbridge.

171. STONE, C.J. (September 1988). Large-sample inference for logspline models.

172. ADLER, R.J. and EPSTEIN, R. (September 1988). Intersection local times for infinite systems of planar brownianmotions and for the brownian density process.

173. MILLAR, P.W. (October 1988). Optimal estimation in the non-parametric multiplicative intensity model.

174. YOR, M. (October 1988). Interwinings of Bessel processes.

175. ROJO, J. (October 1988). On the concept of tail-heaviness.

176. ABRAHAMS, DlM. and RIZZARDI, F. (September 1988). BLSS - The Berkeley interactive statistical system:An overview.

7-

177. MILLAR, P.W. (October 1988). Gamma-funnels in the domain of a probability, with statistical implications.

178. DONOHO, D.L. and LIU, R.C. (October 1988). Hardest one-dimensional subproblems.

179. DONOHO, D.L. and STARK, P.B. (October 1988). Recovery of sparse signal when the low frequency information ismissing.

180. FREEDMAN, DA. and PITMAN, JA. (Nov. 1988). A measure which is singular and uniformly locally uniform.(Revision of Tech Report No. 163).

181. DOKSUM, KA. and HOYLAND, ARNLJOT (Nov. 1988, revised Jan. 1989, Aug. '89). Models for variable stressaccelerated life testing experiments based on Wiener processes and the inverse Gaussian distribution.

182. DALANG, R.C., MORTON, A. and WILL1NGER, W. (November 1988). Equivalent martingale measures andno-arbitrage in stochastic securities market models.

183. BERAN, R. (November 1988). Calibrating prediction regions.

184. BARLOW, M.T., P1iTMAN, J. and YOR, M. (Feb. 1989). On Walsh's Brownian Motions.

185. DALANG, R.C. and WALSH, J.B. (Dec. 1988). Almost-equivalence of the germ-field Markov property and the sharpMarkov propt of the Brownian sheet.

186. HESSE, C.H. (Dec. 1988). Level-Crossing of integrated Ornstein-Uhlenbeck processes

187. NEVEU, J. and PriMAN, J.W. (Feb. 1989). Renewal property of the extrema and tree property of the excursionof a one-dimensional brownian motion.

188. NEVEU, J. and PiTMAN, J.W. (Feb. 1989). The branching process in a brownian excursion.

189. PiTMAN, J.W. and YOR, M. (Mar. 1989). Some extensions of the arcsine law.

190. STARK, PB. (Dec. 1988). Duality and discretization in linear inverse problems.

191. LEHMANN, E.L. and SCHOLZ, F.W. (Jan. 1989). Ancillarity.

192. PEMANTLE, R. (Feb. 1989). A time-dependent version of P6lya's um.

193. PEMANTLE, R. (Feb. 1989). Nonconvergence to unstable points in urn models and stochastic approximations.

194. PEMANTLE, R. (Feb. 1989, revised May 1989). When are touchpoints limits for generalized P6lya urns.

195. PEMANTLE, R. (Feb. 1989). Random walk in a random environment and first-passage percolation on trees.

196. BARLOW, M., P1TMAN, J. and YOR, M. (Feb. 1989). Une extension multidimensionnefle de la loi de l'arc sinus.

197. BREIMAN, L. and SPECTOR, P. (Mar. 1989). Submodel selection and evaluation in regression- the X-random case.

198. BREIMAN, L., TSUR, Y. and ZEMEL, A. (Mar. 1989). A simple estimation procedure for censoredregression models with known error distribution.

199. BRILLINGER, D.R. (Mar. 1989). Two papers on bilinear systems: a) A study of second- and third-order spectralprocedures and maximum likelihood identification of a bilinear system. b) Some statistical aspects of NMR spectros-copy, Actas del 2° congreso lantinoamericano de probabilidad y estadistica matematica, Caracas, 1985.

200. BRILLINGER, D.R. (Mar. 1989). Two papers on higher-order spectra: a) Parameter estimation for nonGaussian processesvia second and third order spectra with an application to some endocrine data. b) Some history of the study of higher-order moments and specta

201. DE LA PENA, V. and KLASS, MJ. (April 1989). L bounds for quadratic forms of independent random variables.

202. FREEDMAN, DA. and NAVIDI, W.C. (April 1989). Testing the independence of competing risks.

203. TERDIK, G. (May 1989). Bilinear state space realization for polynomial stochastic systems.

204. DONOHO, D.L. and JOHNSTONE, LM. (May 1989). Minimax risk over Ip-Balls.

205. PEMANTLE, R., PROPP, J. and ULLMAN, D. (May 1989). On tensor powers of integer programs.

- 8-

206. MILASEVIC, P. and NOLAN, D. (May 1989). Estimation on the sphere: A geometric approach.

207. SPEED, T.P. and YU, B. (July 1989). Stochastic complexity and model selection: normal regression.

208. DUBINS, L.E. (June 1989). A group decision device: Its pareto-like optimality.

209. BREIMAN, L. (July 1989). Fitting additive models to regression data.

210. PEMANTLE, R. (July 1989). Vertex-reinforced random walk.

211. LE CAM, L. (August 1989). On measurability and convergence in distribution.

212. FELDMAN, R.E. (July 1989). Autoregressive processes and first-hit probabilities for randomized random walks.

213. DONOHO, D.L., JOHNSTONE, IM., HOCH, J.C. and STERN, A.S. (August 1989). Maximum entropyand the nearly black object.

Copies of these Reports plus the most recent additions to the Technical Report series are available from the Statistics Depart-ment technical typist in room 379 Evans Hall or may be requested by mail from:

Department of StatisticsUniversity of CalifomiaBerkeley, California 94720

Cost: $1 per copy.