Springer Series in Statistics

Advisors: P. Bickel, P. Diggle, s. Fienberg, K. Krickeberg, 1. Olkin, N. Wermuth, s. Zeger

Springer Science+Business Media, LLC

Springer Series in Statistics

Andersen/Borgan/Gill/Keiding: Statistical Models Based on Counting Processes. Andrews/Herzberg: Data: A Collection of Problems from Many Fields for the Student

and Research Worker. Anscombe: Computing in Statistical Science through APL. Berger: Statistical Decision Theory and Bayesian Analysis, 2nd edition. Bolfarine/Zacks: Prediction Theory for Finite Populations. Bremaud: Point Processes and Queues: Martingale Dynamics. Brockwell/Davis: Time Series: Theory and Methods, 2nd edition. DaleylVere-Jones: An Introduction to the Theory of Point Processes. Dzhaparidze: Parameter Estimation and Hypothesis Testing in Spectral Analysis of

Stationary Time Series. Fahrmeir/Tutz: Multivariate Statistical Modelling Based on Generalized Linear

Models. Farrell: Multivariate Calculation. Federer: Statistical Design and Analysis for Intercropping Experiments. FienberglHoaglin/KruskallTanur (Eds.): A Statistical Model: Frederick Mosteller's

Contributions to Statistics, Science and Public Policy. Fisher/Sen: The Collected Works of Wassily Hoeffding. Good: Permutation Tests: A Practical Guide to Resampling Methods for Testing

Hypotheses. GoodmanlKruskal: Measures of Association for Cross Classifications. Grandell: Aspects of Risk Theory. Haberman: Advanced Statistics, Volume I: Description of Populations. Hall: The Bootstrap and Edgeworth Expansion. Hardie: Smoothing Techniques: With Implementation in S. Hartigan: Bayes Theory. Heyer: Theory of Statistical Experiments. Huet/Bouvier/GruetlJolivet: Statistical Tools for Nonlinear Regression: A Practical

Guide with S-PLUS Examples. Jolliffe: Principal Component Analysis. KolenlBrennan: Test Equating: Methods and Practices. Kotz/Johnson (Eds.): Breakthroughs in Statistics Volume I. Kotz/Johnson (Eds.): Breakthroughs in Statistics Volume II. Kres: Statistical Tables for Multivariate Analysis. Le Cam: Asymptotic Methods in Statistical Decision Theory. Le Cam/Yang: Asymptotics in Statistics: Some Basic Concepts. Longford: Models for Uncertainty in Educational Testing. Manoukian: Modem Concepts and Theorems of Mathematical Statistics. Miller, Jr.: Simultaneous Statistical Inference, 2nd edition. Mosteller/Wallace: Applied Bayesian and Classical Inference: The Case of The

Federalist Papers.

(continued after index)

Jun Shao Dongsheng Tu

The Jackknife and Bootstrap

, Springer

Jun Shao Dongsheng Tu Department of Statistics University of Wisconsin, Madison 1210 West Dayton Street Madison, WI 53706-1685 USA

Institute of System Science Academia Sinica Beijing, 100080 People's Republic of China

With 4 figures.

Library of Congress Cataloging-in-Publication Data Shao, Jun.

The jackknife and bootstrap / Jun Shao, Dongsheng Tu. p. cm. - (Springer series in statistics)

Includes bibliographical references and index. ISBN 978-1-4612-6903-8 ISBN 978-1-4612-0795-5 (eBook) DOI 10.1007/978-1-4612-0795-5 1. Jackknife (Statistics). 2. Bootstrap (Statistics).

3. Resampling (Statistics). 4. Estimation theory. 1. Tu, Dongsheng. II. Title. III. Series. QA276.6.S46 1995 519.5' 44-dc20 95-15074

Printed on acid-free paper.

© 1995 Springer Science+Business Media New York Originally published by Springer-Verlag New York, Inc in 1995 Softcover reprint ofthe hardcover Ist edition 1995 Ali rights reserved. This work may not be translated or copied in whole or in part without the written permission of the publisher Springer Science+Business Media, LLC, except for brief excerpts in connection with reviews or scholarly analysis. Use in connection with any form of information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereaf­ter developed is forbidden. The use of general descriptive names, trade names, trademarks, etc., in this publication, even if the former are not especially identified, is not to be taken as a sign that such names, as understood by the Trade Marks and Merchandise Marks Act, may accordingly be used freely byanyone.

Production managed by Hal Henglein; manufacturing supervised by Joe Quatela. Photocomposed pages prepared from the authors' LaTeX file.

9 8 7 6 5 4 3 2 (Second corrected printing, 1996)

ISBN 978-1-4612-6903-8 SPIN 10544454

To Guang and Shurong

Preface

The jackknife and bootstrap are the most popular data-resampling meth­ods used in statistical analysis. The resampling methods replace theoreti­cal derivations required in applying traditional methods (such as substitu­tion and linearization) in statistical analysis by repeatedly resampling the original data and making inferences from the resamples. Because of the availability of inexpensive and fast computing, these computer-intensive methods have caught on very rapidly in recent years and are particularly appreciated by applied statisticians.

The primary aims of this book are

(1) to provide a systematic introduction to the theory of the jackknife, the bootstrap, and other resampling methods developed in the last twenty years;

(2) to provide a guide for applied statisticians: practitioners often use (or misuse) the resampling methods in situations where no theoretical confirmation has been made; and

(3) to stimulate the use of the jackknife and bootstrap and further devel­opments of the resampling methods.

The theoretical properties of the jackknife and bootstrap methods are studied in this book in an asymptotic framework. Theorems are illustrated by examples. Finite sample properties of the jackknife and bootstrap are mostly investigated by examples and/or empirical simulation studies. In addition to the theory for the jackknife and bootstrap methods in problems with independent and identically distributed (Li.d.) data, we try to cover, as much as we can, the applications of the jackknife and bootstrap in various complicated non-Li.d. data problems.

Chapter 1 introduces some basic ideas and motivations for using the jackknife and bootstrap. It also describes the scope of our studies in this book. Chapters 2 and 3 contain general theory for the jackknife and the bootstrap, respectively, which sets up theoretical fundamentals. Some tech­nical tools are introduced and discussed in these two chapters for readers

vii

Vlll Preface

interested in theoretical studies. Beginning with Chapter 4, each chapter focuses on an important topic concerning the application of the jackknife, the bootstrap, and other related methods. Chapter 4 studies bootstrap con­fidence sets in some depth: we consider various bootstrap confidence sets developed in recent years and provide asymptotic and empirical compar­isons. Bootstrap hypothesis testing is also studied in Chapter 4. Chapter 5 discusses some computational aspects of the jackknife and bootstrap meth­ods. Chapter 6 considers sample survey problems, one of the non-i.i.d. data problems in which the use of the resampling methods (e.g., the jackknife and balanced repeated replication) has a long history. Chapter 7 focuses on applications of the jackknife and bootstrap to linear models, one of the most useful models in statistical applications. Chapter 8 contains some recent developments of the jackknife and bootstrap in various other impor­tant statistical fields such as nonlinear regression, generalized linear models, Cox's regression, nonparametric density estimation, nonparametric regres­sion, and multivariate analysis. Applications of the jackknife and bootstrap for dependent data (time series) are studied in Chapter 9. The last chapter introduces two resampling methods that are generalizations of the boot­strap, namely the Bayesian bootstrap and the random weighting. Except for the first chapter, each chapter ends with conclusions and discussions.

Some useful asymptotic results that are often cited in this book are provided in Appendix A. A list of notation is given in Appendix B.

Some knowledge of mathematical statistics (with a standard textbook such as Bickel and Doksum, 1977) is assumed. The reader should be famil­iar with concepts such as probability, distribution, expectation, estimators, bias, variance, confidence sets, and hypothesis tests. For reading Chapters 6-9, some knowledge of the fields under consideration is required: sample surveys for Chapter 6; linear models for Chapter 7; nonlinear models, gen­eralized linear models, nonparametric regression, and multivariate analysis for the respective sections of Chapter 8; and time series for Chapter 9. Some knowledge of prior and posterior distributions in Bayesian analysis is needed for reading Chapter 10. The mathematical level of the book is flexible: a practical user with a knowledge of calculus and a notion of vec­tors and matrices can understand all of the basic ideas, discussions, and recommendations in the book by skipping the derivations and proofs (we actually omitted some difficult proofs); with a knowledge of advanced cal­culus, matrix algebra, and basic asymptotic tools in mathematical statistics (Appendix A and Chapter 1 of Serfling, 1980), one can fully understand the derivations and most of the proofs. A few places (e.g., Section 2.2 and part of Sections 2.4 and 3.3) involving more advanced mathematics (such as real analysis) can be skipped without affecting the reading of the rest of the book.

The Edgeworth and Cornish-Fisher expansions are very important tools

Preface ix

in studying the accuracy of the bootstrap distribution estimators and boot­strap confidence sets. However, the derivations and rigorous proofs of these expansions involve difficult mathematics, which can be found in a recent book by Hall (1992d) (some special cases can be found in Appendix A). Thus, we only state these expansions (with the required regularity condi­tions) when they are needed, without providing detailed proofs. This does not affect the understanding of our discussions.

Although conceived primarily as a research monograph, the book is suitable for a second-year graduate level course or a research seminar. The following are outlines for various possible one-semester courses.

I

(I) AN INTRODUCTION TO JACKKNIFE AND BOOTSTRAP (non-theoretical)

Chapter 1, Sections 2.1 and 2.3, Sections 3.4 and 3.5, Chapter 4 (except Sections 4.2 and 4.3), and Chapter 5. Technical proofs should be skipped. If there is time, include Sections 6.1, 6.2, 6.3, 7.1, 7.2, and 7.3.

(II) JACKKNIFE AND BOOTSTRAP FOR COMPLEX DATA (non-theoretical)

Chapter 1, Chapter 6 (except Section 6.4), Chapter 7 (except Section 7.5), Chapter 8, and Chapter 9. Technical proofs should be skipped. If there is time, include some materials from Chapter 5 or Section 1O.l.

(III) THEORY OF JACKKNIFE AND BOOTSTRAP (theoretical)

Chapters 1-5. If there is time, include some materials from Chapters 6 and 7 or Chapter 10.

(IV) JACKKNIFE AND BOOTSTRAP FOR COMPLEX DATA (theoretical)

Chapters 1, 6-9. If there is time, include Chapter 5 or Chapter 10.

Of course, one may combine either (I)-(II) or (III)-(IV) for a two­semester course.

This book is essentially based on the authors' lecture notes for graduate level courses taught at Purdue University in 1988, at the University of Waterloo in 1989, and at the University of Ottawa in 1991 and 1992. We are very grateful to students and colleagues who provided helpful comments. Special thanks are due to C. R. Rao, who provided some critical comments on a preliminary draft of this book; J. N. K. Rao, who read part of the manuscript and provided useful suggestions; and anonymous referees and Springer-Verlag Production and Copy Editors, who helped to improve the presentation. We also would like to express our appreciation to the National Sciences and Engineering Research Council of Canada for support during the writing of the book.

Madison and Ottawa May, 1995

J. Shao D. Tu

Contents

Preface

Chapter 1. Introduction

1.1 Statistics and Their Sampling Distributions

1.2 The Traditional Approach

1.3 The Jackknife . . . . . . . . . . .

1.4 The Bootstrap ......... .

1.5 Extensions to Complex Problems

1.6 Scope of Our Studies . . . . . . .

Chapter 2. Theory for the Jackknife

2.1 Variance Estimation for Functions of Means

2.2

2.3

2.1.1 Consistency ....... .

2.1.2 Other properties ...... .

2.1.3 Discussions and examples ..

Variance Estimation for Functionals

2.2.1 Differentiability and consistency

2.2.2 Examples...........

2.2.3 Convergence rate . . . . . . .

2.2.4 Other differential approaches

The Delete-d Jackknife ....

2.3.1 Variance estimation .

2.3.2 Jackknife histograms.

2.4 Other Applications . . .

2.4.1 Bias estimation.

2.4.2 Bias reduction .

xi

vii

1

1

2 4 9

17

19

23

23

24

28

29 32

33

37

42

44 49 50

55

60 61

64

xii

2.4.3 Miscellaneous results .

2.5 Conclusions and Discussions.

Chapter 3. Theory for the Bootstrap 3.1 Techniques in Proving Consistency

3.1.1 Bootstrap distribution estimators.

3.1.2 Mallows' distance .....

3.1.3 Berry-Esseen's inequality

3.1.4 Imitation ..

3.1.5 Linearization

3.1.6 Convergence in moments

3.2 Consistency: Some Major Results.

3.2.1 Distribution estimators

3.2.2 Variance estimators ..

3.3 Accuracy and Asymptotic Comparisons

3.3.1

3.3.2

3.3.3

3.3.4

3.3.5

Convergence rate . . . . . . . . .

Asymptotic minimaxity .....

Asymptotic mean squared error .

Asymptotic relative error

Conclusions . . . . .

3.4 Fixed Sample Performance

3.4.1 Moment estimators .

3.4.2 Distribution estimators

3.4.3 Conclusions.

3.5 Smoothed Bootstrap

3.5.1 Empirical evidences and examples

3.5.2 Sample quantiles

3.5.3 Remarks.

3.6 Nonregular Cases

3.7 Conclusions and Discussions .

Chapter 4. Bootstrap Confidence Sets and Hypothesis Tests 4.1 Bootstrap Confidence Sets . . .

4.1.1 The bootstrap-t

4.1.2 The bootstrap percentile.

Contents

68 69

71

72

72 73

74 76 78 79

80 80 86 91 91

97

99 102

104

104

105

108

112 113 113 116 117 118 127

129 129

131

132

Contents xiii

4.1.3 The bootstrap bias-corrected percentile ..... .. 133

4.1.4 The bootstrap accelerated bias-corrected percentile. 135

4.1.5 The hybrid bootstrap 140

4.2 Asymptotic Theory. 141

4.2.1 Consistency. 141

4.2.2 Accuracy . . 144

4.2.3 Other asymptotic comparisons 152

4.3 The Iterative Bootstrap and Other Methods. 155

4.3.1 The iterative bootstrap ........ 155

4.3.2 Bootstrap calibrating ......... 160

4.3.3 The automatic percentile and variance stabilizing. 161

4.3.4 Fixed width bootstrap confidence intervals 164

4.3.5 Likelihood based bootstrap confidence sets 165

4.4 Empirical Comparisons. . . . . . . . . . . . . . . . 166

4.4.1 The bootstrap-t, percentile, BC, and BCa . 166

4.4.2 The bootstrap and other asymptotic methods 170

4.4.3 The iterative bootstrap and bootstrap calibration. 173

4.4.4 Summary...... 176

4.5 Bootstrap Hypothesis Tests 176

4.5.1 General description. 177

4.5.2 Two-sided hypotheses with nuisance parameters 179

4.5.3 Bootstrap distance tests . . . 182

4.5.4 Other results and discussions 184

4.6 Conclusions and Discussions. . . 188

Chapter 5. Computational Methods

5.1 The Delete-l Jackknife ....

5.1.1 The one-step jackknife

5.1.2 Grouping and random subsampling .

5.2 The Delete-d Jackknife ....

5.2.1 Balanced subsampling

5.2.2 Random subsampling

5.3 Analytic Approaches for the Bootstrap .

5.3.1 The delta method ....

5.3.2 Jackknife approximations

190

190

191

195

197

197

198

200

201

202

xiv

5.4

5.5

5.3.3 Saddle point approximations ...

5.3.4 Remarks...............

Simulation Approaches for the Bootstrap

5.4.1 The simple Monte Carlo method

5.4.2 Balanced bootstrap resampling

5.4.3 Centering after Monte Carlo

5.4.4 The linear bootstrap . . . . .

5.4.5 Antithetic bootstrap resampling

5.4.6 Importance bootstrap resampling .

5.4.7 The one-step bootstrap

Conclusions and Discussions . .

Contents

203

205

206

207

211

215

219

221

223

228

230

Chapter 6. Applications to Sample Surveys

6.1 Sampling Designs and Estimates

232

232

238

238

241

244

246

6.2 Resampling Methods ...... .

6.2.1 The jackknife ...... .

6.2.2 The balanced repeated replication

6.2.3 Approximated BRR methods

6.2.4 The bootstrap

6.3 Comparisons by Simulation 251

6.4 Asymptotic Results. . . . . 258

6.4.1 Assumptions . . . . 258

6.4.2 The jackknife and BRR for functions of averages 260

6.4.3 The RGBRR and RSBRR for functions of averages. 264

6.4.4 The bootstrap for functions of averages '" 267

6.4.5 The BRR and bootstrap for sample quantiles 268

6.5 Resampling Under Imputation

6.5.1 Hot deck imputation ..

6.5.2 An adjusted jackknife .

6.5.3 Multiple bootstrap hot deck imputation

6.5.4 Bootstrapping under imputation

6.6 Conclusions and Discussions ...... .

Chapter 7. Applications to Linear Models

7.1 Linear Models and Regression Estimates

7.2 Variance and Bias Estimation ..... .

270

271

273

277

278

281

283

283

285

Contents

7.2.1 Weighted and unweighted jackknives

7.2.2 Three types of bootstraps ..... .

7.2.3 Robustness and efficiency ..... .

7.3 Inference and Prediction Using the Bootstrap

7.3.1

7.3.2

7.3.3

7.3.4

Confidence sets . . . . . . . . . . .

Simultaneous confidence intervals .

Hypothesis tests

Prediction .

7.4 Model Selection . .

7.4.1 Cross-validation

7.4.2 The bootstrap .

7.5 Asymptotic Theory ...

7.5.1 Variance estimators

7.5.2 Bias estimators ...

7.5.3

7.5.4

7.5.5

Bootstrap distribution estimators .

Inference and prediction

Model selection . . . .

7.6 Conclusions and Discussions.

Chapter 8. Applications to Nonlinear, Nonparametric,

xv

285

289

292

295

295

298

301

303

306

307

311

313

313

318

320

324

326

329

and Multivariate Models 331 8.1 Nonlinear Regression. . . . . . . . . . . . . . . . . 331

8.2

8.1.1 Jackknife variance estimators ....... .

8.1.2 Bootstrap distributions and confidence sets

8.1.3 Cross-validation for model selection

Generalized Linear Models. . . . . .

8.2.1

8.2.2

8.2.3

Jackknife variance estimators

Bootstrap procedures ....

Model selection by bootstrapping .

8.3 Cox's Regression Models ...... .

8.3.1 Jackknife variance estimators

8.3.2 Bootstrap procedures ....

8.4 Kernel Density Estimation. . . . . .

8.4.1 Bandwidth selection by cross-validation

8.4.2 Bandwidth selection by bootstrapping

8.4.3 Bootstrap confidence sets . . . . . . .

333

335

337

338

340

341

343

345

346

349

350

351

353

356

xvi

8.5 Nonparametric Regression .......... .

8.5.1 Kernel estimates for fixed design .. .

8.5.2 Kernel estimates for random regressor

8.5.3 Nearest neighbor estimates

8.5.4 Smoothing splines . . . . .

8.6 Multivariate Analysis ....... .

8.6.1

8.6.2

8.6.3

8.6.4

Analysis of covariance matrix

Multivariate linear models.

Discriminant analysis ...

Factor analysis and clustering .

8.7 Conclusions and Discussions ..... .

Contents

360

360

364

366

370

373

373

376

379

382

384

Chapter 9. Applications to Time Series and Other Dependent Data 386

387

392

394

395

397

400

401

403

406

407

407

410

411

413

414

9.1 m-Dependent Data .... .

9.2 Markov Chains ...... .

9.3 Autoregressive Time Series

9.3.1 Bootstrapping residuals

9.3.2 Model selection

9.4 Other Time Series ..

9.4.1 ARMA(p, q) models

9.4.2 Linear regression with time series errors

9.4.3 Dynamical linear regression .. .

9.5 Stationary Processes .......... .

9.5.1 Moving block and circular block

9.5.2 Consistency of the bootstrap

9.5.3 Accuracy of the bootstrap.

9.5.4 Remarks ....... .

9.6 Conclusions and Discussions.

Chapter 10. Bayesian Bootstrap and Random Weighting 416

10.1 Bayesian Bootstrap. . . . . . . . . . . . . . . . . . . . . 416

10.1.1 Bayesian bootstrap with a noninformative prior. 417

10.1.2 Bayesian bootstrap using prior information 420

lO.1.3 The weighted likelihood bootstrap 422

10.1.4 Some remarks. . . . . . . . . . . . 424

Contents xvii

10.2 Random Weighting 425

10.2.1 Motivation 425

10.2.2 Consistency 427

10.2.3 Asymptotic accuracy. 429

10.3 Random Weighting for Functionals and Linear Models 434

10.3.1 Statistical functionals ....... 434

10.3.2 Linear models. . . . . . . . . . . . 437

10.4 Empirical Results for Random Weighting 440

10.5 Conclusions and Discussions.

Appendix A. Asymptotic Results A.1 Modes of Convergence . . . .

A.2 Convergence of Transformations.

A.3 0(·), 0(·), and Stochastic O( .), 0(·) A.4 The Borel-Cantelli Lemma. . . . . .

A.5 The Law of Large Numbers .....

A.6 The Law of the Iterated Logarithm .

A.7 Uniform Integrability ....

A.8 The Central Limit Theorem

A.9 The Berry-Esseen Theorem

A.10 Edgeworth Expansions ...

A.ll Cornish-Fisher Expansions.

Appendix B. Notation

References

Author Index

Subject Index

445

447 447

448

448

449

449

450

450

451

451

452

454

455

457

493

499