This article was downloaded by: [University of Kiel]On: 24 October 2014, At: 21:21Publisher: RoutledgeInforma Ltd Registered in England and Wales Registered Number: 1072954 Registered office: Mortimer House,37-41 Mortimer Street, London W1T 3JH, UK

Research Quarterly for Exercise and SportPublication details, including instructions for authors and subscription information:http://www.tandfonline.com/loi/urqe20

Making Bootstrap Statistical Inferences: A TutorialWeimo Zhu aa Division of Health, Physical Education, and Recreation , Wayne State University , USAPublished online: 26 Feb 2013.

To cite this article: Weimo Zhu (1997) Making Bootstrap Statistical Inferences: A Tutorial, Research Quarterly for Exercise andSport, 68:1, 44-55, DOI: 10.1080/02701367.1997.10608865

To link to this article: http://dx.doi.org/10.1080/02701367.1997.10608865

PLEASE SCROLL DOWN FOR ARTICLE

Taylor & Francis makes every effort to ensure the accuracy of all the information (the “Content”) containedin the publications on our platform. However, Taylor & Francis, our agents, and our licensors make norepresentations or warranties whatsoever as to the accuracy, completeness, or suitability for any purpose of theContent. Any opinions and views expressed in this publication are the opinions and views of the authors, andare not the views of or endorsed by Taylor & Francis. The accuracy of the Content should not be relied upon andshould be independently verified with primary sources of information. Taylor and Francis shall not be liable forany losses, actions, claims, proceedings, demands, costs, expenses, damages, and other liabilities whatsoeveror howsoever caused arising directly or indirectly in connection with, in relation to or arising out of the use ofthe Content.

This article may be used for research, teaching, and private study purposes. Any substantial or systematicreproduction, redistribution, reselling, loan, sub-licensing, systematic supply, or distribution in anyform to anyone is expressly forbidden. Terms & Conditions of access and use can be found at http://www.tandfonline.com/page/terms-and-conditions

Research Quarterly for Exercise andSport© 1997 bytheAmerican Alliance for Health,Physical Education, Recreation and DanceVol. 68, No.1,pp.44-55

Making Bootstrap Statistical Inferences: ATutorial

WeimoZhu

Bootstrapping is a computer-intensive statistical technique in which extensive computational procedures are heavily dependent onmodern high-speed digital computers. The payofffor such intensive computations is freedom from two major limitingfactors thathave dominated classical statistical theory since its beginning: the assumption that the data conform to a bell-shaped curve, andthe need to focus on statistical measures whose theoreticalproperties can be analyzed mathematically. The name "bootstrap" wasderived from an old saying about pulling oneselfup by one's own bootstraps. In this case, bootstrapping means redrawingsamples randomly from the original sample with replacement. The key idea, computations, advantages, limitations, and applica­tion potential ofbootstrapping in the field ofphysical education and exercise science are introduced and illustrated using a set ofnational physical fitness testing data. Finally, an example ofa bootstrapping application is provided. Through a step-by-stepapproach, the development and implementation of the bootstrap statistical inference are illustrated.

Key words: methodology, computer-intensive statistics,nonparametric statistics

H oWmany sit-ups can an average American 10-year­old girl do in 1 min? To what degree do females'

body-mass indexes correlate with their percentage ofbody fat? Researchers try to answer these kinds ofpopulation-related questions in their professionalpractices. In real life situations, however, they rarelytry to answer these questions by studying the popula­tion directly due to the population size (e.g., all 10­year-old American girls) and the cost of such a study.Instead, researchers draw samples from the popula­tion and try to make inferences about the populationbased on sample statistics, such as means and standarddeviations. The use of sample statistics to infer thevalue of a population parameter is called "point esti­mation."

In reality, however, the value of a sample statisticwill usually not be exactly the same as the true popu­lation value because of sampling error. Therefore, itis necessary to qualify an estimation to indicate thegeneral magnitude of this error, or the accuracy, ofthe estimation. Often, this is done through construct-

Submitted: January 7, 1994Accepted: July29, 1996

Weimo Zhuis with theDivision of Health, Physical Education, andRecreation at Wayne State University.

44

ing an "interval estimation," which describes a rangeofvalues within which a population parameter is likelyto lie. The 95% confidence interval, in which the in­terval estimate is constructed by using ±1.96 standarderrors from the statistic employed, is perhaps the mostcommonly used interval estimate. There is, however,an important assumption underlying this commonpractice (i.e., the distribution of the sample statistic,known also as the sampling distribution, must be "nor­mal" or bell shaped). The assumption, therefore, isknown also as the assumption of normality. In fact,most of the currently used "classical" statistics (e.g.,Pearson's product-moment correlation coefficient,t test, and analysis ofvariance [ANOVA]), which weredeveloped at the end of the last century and the earlypart of this century when computing power was slowand expensive, are based on certain, often unverifi­able, assumptions. In statistics, those based on suchassumptions are referred to as parametric statistics.

Data collected in practice, however, often do notmeet the assumptions. For example, the data usuallyare not distributed normally when the sample size issmall or the original population itself is not distrib­uted normally. In this situation, applying parametricstatistics depends on the central limit theorem (i.e.,if one repeats the drawing of samples from the popu­lation, the sampling distribution of a statistic ap­proaches a bell-shaped curve).

In reality, however, it is almost impossible for prac­titioners or researchers to draw many repeatedsamples from a population. Therefore, parameters ofsampling distributions must be estimated. This means

ROES: March 1997

Dow

nloa

ded

by [

Uni

vers

ity o

f K

iel]

at 2

1:21

24

Oct

ober

201

4

that traditional parametric statistical inference re­quires both a distribution assumption about a samplingdistribution and a readily available formula to calculatethe parameters of the distribution. In fact, that is oneof the major reasons why statisticians in the past devotedmost of their time to developing new formulas. Still, onlylimited statistics are available, because formulas for otherstatistical procedures have not been developed yet or aretoo difficult to derive mathematically. For example, re­searchers often use only the mean, median, and modeto estimate central tendency, although other statistics,such as the trimmed mean, may be more appropriatein certain circumstances. Furthermore, studies (DiCiccio& Romano, 1989; Efron, 1987) have shown that only inrare cases do parametric assumptions allow for an exactestimation ofsampling distributions, especially when thesample size is small (conventionally n ~ 30). Often, thereare three alternatives when parametric statistics are notappropriate.

The first is to make statistical inferences usingparametric statistics, disregarding the fact that the as­sumptions of the statistics have been violated but be­lieving the results will be robust. In other words,accurate inferences may be achieved even if the as­sumptions have been violated. The commonly used ttest and ANOVA, for example, are quite robust, withrespect to violation of the normality assumption. Themajor drawback of this alternative is that the degree ofassumption violation is often difficult to measure, al­though various robust indexes (see Hill & Dixon, 1982)have been developed.

The second alternative is to apply a transformationin which scores are changed systematically (e.g., takingthe square root ofeach of the scores, so that the assump­tions can be met). The major shortcomings are eitherthat the transformation is not necessary (e.g., the useofvariance-stabilizing transformations has little effect onthe significance level and power of the F test [Budescu& Appelbaum, 1981]) or that the transformation maybe not helpful at all (e.g., no transformation will makethe data more suitable for ANOVA if the means of treat­ment levels are approximately equal, but variances of theerror effects are heterogeneous [Kirk, 1982]). Further­more, the transformation is sometimes limited by its ownassumptions, and results based on the new scale is some­times difficult to interpret.

The third alternative is to make inferences usingclassical nonparametric statistics. The sign test, thematched-pair Wilcoxon test, and the Kruskal-Wallistest are a few examples in this category. The majordrawback of using nonparametric statistics is thatmuch information about the data is lost in the com­puting process (i.e., taking into account only the or­dering of observations but not their values), andthereafter, the inference derived from the nonparamet­ric statistic may sometimes lack the statistical power

ROES: March 1997

Zhu

needed. Recently developed bootstrapping statisticsprovide another, new alternative.

Bootstrapping

Bootstrapping andJackknife

Bootstrapping, invented by Efron (1979), was basedon the idea of thejackknife technique (Quenouille, 1949;Tukey, 1958), which is used to reduce the bias of an esti­mator. The name "bootstrap" was derived from an oldsaying about pulling oneself up by one's own bootstraps,reflecting the fact that the one available sample gives riseto many others. Statistics in both techniques, known alsoas resampling techniques (Efron, 1982), are developedbased on subsamples from a sample originally drawnfrom the population. However, jackknife resampling isconducted in a more systematic way: Each observationis omitted, in turn, to generate subsamples. If, for ex­ample, there are 20 observations in a sample, to gener­ate the first jackknife sample statistic (e.g., the mean),omit the first observation from the sample and computethe mean based on the remaining observations; to gen­erate the secondjackknife mean, bring back the first ob­servation, but omit the second one. Repeat this for allthe observations until a total of 20 jackknife means arecomputed.Jackknife estimations then can be determinedbased on these jackknife means. Bootstrap resampling,in contrast, is conducted randomly with replacement andmany more repetitions, described in detail later in thisstudy. Although jackknife resampling does not place aheavy burden on computation, quite often the jackknifewill not work as well as bootstrapping in practice (Efron& Tibshirani, 1993). For greater detail ofjackknifing andits relationship to bootstrapping, refer to Miller (1974)and Efron and Tibshirani (1993).

Bootstrapping andComputer-Intensive Statistics

The development of bootstrapping cannot be sepa­rated from the development of modern digital com­puters, because it is impossible to implement thebootstrapping idea, which requires many resamplings,without the assistance of modern computer power. Infact, it is the reason that bootstrapping, together withrandomization or permutation and Monte Carlo tests, isreferred to as computer-intensive statistics in the litera­ture (Diaconis & Efron, 1983; Noreen, 1989). Computer­intensive statistics are free from two major limiting factorsthat have dominated classical statistical theory since itsbeginning: the assumption that the data conform to abell-shaped curve, and the need to focus on statisticalmeasures whose theoretical properties can be analyzedmathematically (Diaconis & Efron, 1983).

45

Dow

nloa

ded

by [

Uni

vers

ity o

f K

iel]

at 2

1:21

24

Oct

ober

201

4

Zhu

Bootstrapping Samples andSampling Distribution

As with commonly used parametric inferential sta­tistics, bootstrapping also is based on a sampling distri­bution. The bootstrap sampling distribution, however,is not developed from many repeated samples from thepopulation or any analytic formulas. A bootstrap sam­pling distribution is developed by redrawing a randomsample from a population. More specifically, to developa bootstrap sampling distribution, first draw a samplerandomly from a population. Instead of drawing suc­ceeding samples repeatedly from the population orcomputing parameters of the sampling distributionbased on existing analytic formulas, redraw many

subsamples, called bootstrap samples, randomly withreplacement from the sample. Thus, bootstrapping hereactually means redrawing samples randomly from theoriginal sample, with replacement. Sample statistics,such as mean and median, are then computed for eachbootstrap sample and called bootstrap statistics. Distri­butions of the bootstrap sample statistics are thebootstrapping sampling distributions of the statistics.Thus, both parametric statistics and bootstrap statisticsestimate the parameters ofa sampling distribution froma sample. Parametric statistics estimate the parametersbased on available analytic formulas, which are restrictedby related distribution assumptions; while bootstrap sta­tistics estimate the parameters based on the bootstrap

Many origiflQl samplesfrom 1hepopulation

Samplestalistlc

sampling diattibUdcnof sample sllil1fatlc

Eattmaled aampHrodWibulion ofsample stallslIc

ParametrICestimation

one 0riCinal samplefrom ttre population

u.J.l1J.u. ._-~~.~.._...~

• • •""""

""'s. L11.LLll1

One CIriglnal sample­from 1Mpopulallon

Figure 1.Differences in developing sampling distributions.

46 ROES: March 1997

Dow

nloa

ded

by [

Uni

vers

ity o

f K

iel]

at 2

1:21

24

Oct

ober

201

4

samples, which are free from these assumptions. Thedifferent steps taken to develop the sampling distribu­tion and the bootstrap sampling distribution are illus­trated in Figure 1.

HowBootstrapping Works

The idea behind bootstrapping is quite simple. Al­though it is impossible to get many samples from a popu­lation (F), it is possible to get repeated samples from apopulation (f) whose distribution approximates thepopulation. Given a sample drawn randomly from thepopulation, the cumulative distribution of the sample,known also as the "empirical distribution function," is theoptimal estimator of the population. Let X = (x., x2' ...,xn) denote a random sample ofsize n from a population.IfF denotes the cumulative distribution function of thepopulation, then F(x) = P(X ~ x), where P(X ~ x) de­notes the probability ofyielding a value less than or equalto x. This allows the cumulative distribution of the sample

Zhu

denoted by F(X(i») to become the maximum likelihoodestimate ofthe population distribution function, F(x):

A i number ofXI'x.; ..., X ~ x.F(x.)=-= -l! n I (1)ro n n '

where i is, in fact, the cumulative frequency and n is thesample size (see Rohatgi, 1984, pp. 234-236, for moredetails) .

The following example illustrates how the F(X(i» isdeveloped and implemented in determining the centraltendency of a population. A small sample of sit-up datawas drawn from the National Children and Youth FitnessStudy (Ross & Gilbert, 1985), which was defined as thepopulationI studied. There were 20 observations in this~ample (Table 1). The maximum value was 57, and theminimum value was 5, M =32.85, SD=14.40. After ar­ranging the data into an order of increasing magnitudeyields X(I) ~ X(2)' ..., ~ x(n)' the cumulative distribution ofthe sample, F(X(i»)' was computed (Table 1) and plotted

Table 1.A sit-up sample data, cumulative distribution function, and related bootstrap samples

Raw data In =20)35 28 18 32 30 5 57 41 50 49 53 44 35 39 25 40 23 11 12 30

Ordered observations xll) ... xln]5 11 12 18 23 25 28 30 32 35 39 40 41 44 49 50 53 57

Frequency 2 2

Cumulativefrequency 2 3 4 5 6 7 9 10 12 13 14 15 16 17 18 19 20

F(x(i)l 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.45 0.50 0.60 0.65 0.70 0.75 0.80 0.85 0.90 0.95 1.00

Frequencies (fl and statistics of 10bootstrap samples

Data: 5 11 12 18 23 25 28 30 32 35 39 40 41 44 49 50 53 57It) 1 1 1 1 1 1 1 2 1 2 1 1 1 1 1 1 1 1

20%Bootstrap Trimmed

sample Mean Median Mean

1 2 1 2 2 3 1 4 3 34.90 35.00 36.082 2 2 2 3 3 4 2 1 28.95 30.00 30.333 3 2 2 1 2 3 2 1 1 2 29.25 31.00 28.584 1 2 3 1 1 1 3 1 1 2 1 2 31.85 37.00 32.755 1 2 1 2 2 1 3 1 1 1 2 1 2 35.30 40.00 36.836 2 1 1 1 3 2 1 1 2 2 1 3 33.60 32.50 34.337 1 1 1 1 1 3 1 4 1 3 1 1 1 36.45 39.00 37.088 5 1 2 2 1 1 1 1 2 3 28.25 24.00 25.509 1 1 1 2 2 2 5 3 1 31.95 35.00 33.0810 3 3 1 2 1 3 2 2 27.70 32.50 27.17

Bootstrap estimate 31.82 33.60 32.17Standard error 3.17 4.71 3.91

ROES: March 1997 47

Dow

nloa

ded

by [

Uni

vers

ity o

f K

iel]

at 2

1:21

24

Oct

ober

201

4

Zhu

eluding frequencies and statistics of resampled obser­vations in these samples.

Based on these samples, bootstrap sample statisticswere calculated. For example, for the first bootstrapsample, M = 34.90. The bootstrap estimates can be com­puted based on t,!le bootstrap sample statistics. The boot­strap estimate (8

B) of the population mean, a, can be

defined as follows':

where B is the number of bootstrap samples (B = 10 inthis case) and S* is the bootstrap sample mean. For thebootstrap estimate of the mean, which is the average ofthe bootstrap sample means, M = 31.82, SD= 3.17 (seeTable 1). Because these bootstrap sample means arereally functioning the same as the real sample means,the mean of the bootstrap sampling distribution canalso be expressed as: 9B = E(X

bo) = u, where Jlis theolStrap

population mean. In other words, the mean of thesampling distribution of bootstrap means, as it is in aparametric sampling distribution (Hays, 1981), is thesame as the population mean. Similarly, the standarddeviation of the bootstrap sampling distribution is thestandard error, or, more accurately, the bootstrap es­timated standard error of the mean. Two other boot-

(Figure 2). For example, the cumulative frequency ofx(2), that is, value 11, was 2, and the correspondingfi'(X(2» was0.10 (2/20), which meant that for this popu­lation the probability ofyielding a value less than or equalto 11 was 0.10.

A nice feature of the cumulative distribution func­tion is that its value has a range from 0 to 1, and each xvalue has its own corresponding cumulative proportion!CP). For example, because the F(X(I» was 0.05 and theF(X(2» was 0.10, the cumulative proportion of the value11, CP(ll)' was in the range: 0.05 < CP(ll):S; 0.10 (see Table1). Based on the cumulative distribution function of thesample and its corresponding cumulative proportions,one can begin the bootstrap sampling process. First,generate a uniform random number between 0 and1, using a random number generator. Then, deter­mine the corresponding value of x according to thecumulative distribution function. Say, for example,the first uniform random number generated was0.6812, which, according the cumulative distributionfunction (Table 1), fell into the CP range of the value40 (0.65 < CP(40) s 0.70) (see Figure 2). The value of40, therefore, was selected, and became the first ob­servation in the first bootstrap example. This processwas repeated 20 times (each bootstrap sample was thesame size as the original sample), and the first boot­strap sample, with 20 observations, was generated. Tensuch bootstrap samples are illustrated in Table 1, in-

1 Be = - ~ e*B "'" i

B i = 1(2)

1

0.9

0.8~1t"e 0.7sJ 0.6

I 0.5

~

ta0.1

00 10 20 30 40 50 60

Sit-Ups(number in 60 sec.)

Figure 2.Cumulative distribution function ofthe sample.

48 ROES: March 1997

Dow

nloa

ded

by [

Uni

vers

ity o

f K

iel]

at 2

1:21

24

Oct

ober

201

4

strap estimates, the median and the 20% trimmedmean also are summarized in Table 1. By using thesame bootstrap samples, other useful statistics (some ofwhich cannot be computed using parametric statistics)can then be computed at the same time, which is an ad­vanced feature of bootstrapping.

Although a bootstrapping sampling distribution, intheory, resembles a parametric sampling distribution,bootstrapping statistics are rarely employed in point es­timation and hypotheses testing", Rather, they are usu­ally used for interval estimations, such as constructingstandard errors and confidence intervals, and other ap­plications related to statistical accuracy, such as correct­ing bias and determining prediction errors (Hjorth,1994). For point estimation and hypotheses testing,other computer-intensive statistical methods, such asrandomization or permutation tests and Monte Carlo,often work better (Noreen, 1989).

Number of Bootstrap Samples

To apply bootstrap statistics, more than 10 bootstrapsamples should be generated. Generally, 50 to 200 boot­strap samples are adequate to develop a bootstrap esti­mate of the standard error (Efron & Tibshirani, 1986).To build bootstrap confidence intervals, however, at least1,000 bootstrap samples should be generated (Efron,1988). To examine the effect of increasing the numberof bootstrap samples (B) on bootstrap statistics, boot­strap statistics were computed when B = 50, B = 200, B =500, B = 1,000, B = 2,000, and B = 5,000, using the samedata set shown in Table 1. The results of thebootstrapping, as well as the "population" parametersand parametric statistics measuring the central tendency,are summarized in Table 2. Overall, there was a trendthat the more bootstrap samples, the better the estima­tion. Improvement of estimation accuracy, however, waslimited, considering the degree of improvement fromthe estimates when B = 10 and B = 5,000. For the sit-updata employed, the parameter statistics provided simi­lar accurate estimations of the true parameters. In fact,as long as a sample is reasonably large (n > 30) and thepopulation distribution is close to a bell-shaped curve,there is little need to replace parametric statistics withbootstrapping.

Original Sample Size andStatisticalPower

Bootstrapping is particularly suited to small datasets, say n ~ 30, because parametric assumptions are of­ten not crucial in large data sets; but bootstrapping alsohas been applied widely to larger and more complexdata sets to which parametric analytic formulas are ei­ther not available or not appropriate. It has been re­ported that in one-sample problems, bootstrapping willfunction well as long as n ~ 8 (Efron, 1995). Generally,

ROES: March 1997

Zhu

the statistical power ofa bootstrap inference, like classi­cal inferential statistics, increases as the sample size in­creases. However, accurate statistical inference based onbootstrapping depends more on how bootstrap samplesare drawn from an original sample. The bootstrapsamples should be drawn from the original sample inthe same way the original sample was drawn from the

Table 2.Theeffect of increasing the number of bootstrapsamples (B)

20%Mean Median Trimmed mean

True parameters

e 33.92 34.00 33.88

Parametric statistics

() 32.85 33.50 33.50

°statistic 3.22' 4.33b _c

Bootstrap estimates

B= 10

~B 31.82 33.60 32.17O'B 3.18 4.71 3.91

B= 508B 32.43 32.46 32.83<\ 3.78 4.78 4.42

B=200

~B 32.64 33.52 33.28O'B 3.07 3.76 3.52

B= 500

~B 32.77 33.21 33.25O'B 3.27 4.01 3.83

B= 1,0008B 32.71 33.33 33.26

°B 3.19 3.88 3.60B= 2,000

~B 32.82 33.39 33.38O'B 3.18 3.78 3.65

B= 5,000

~B 32.90 33.49 33.45O'B 3.13 3.81 3.57

'Thestandard errorwasestimated bythe formula: 0maan=s/(n)"2,where5 is the standard deviation of the sample and n isthesample size.bThe standard errorwasestimated bythe formula: 0m'di,n=(a • b)/3.4641, wherea is thevalue of (n/2 +(3n)"2/2)th observation and bis thevalue of (n/2 • (3n)"2/2)th observation.cNo existing analytic formula was available.

49

Dow

nloa

ded

by [

Uni

vers

ity o

f K

iel]

at 2

1:21

24

Oct

ober

201

4

Zhu

unknown population. In addition, the desired statisticalpower as well as the sample size of a bootstrap statisticalinference also can be computed based on a preliminarybootstrapping data analysis. For more details, refer to arelated chapter by Efron and Tibshirani (1993) and otherrelated descriptions by Hall and Wilson (1991).

Bootstrap Computing

Bootstrap computing is very straightforward: (a)generate a bootstrap sample randomly with replacementfrom the original sample; (b) compute bootstrap samplestatistics and save them; (c) repeat steps (a) and (b) forB times; and (d) compute bootstrap estimates. Theamount ofcomputing required by most bootstrap appli­cations, although considerable, is well within the capa­bilities of today's personal computers. Computinglanguage codes for commonly used statistics have beenincluded in several introductory texts (see Mooney &Duval, 1993, and Noreen, 1989), and most statistical soft­ware packages, such as SAS,SPSS-X,and S'Plus', have thecapacity to conduct bootstrapping with some program­ming. Also, several computer programs specializing inbootstrapping have been developed (see Boomsma,1991, and Lunneborg, 1987).

Advantages of Bootstrapping

The major advantage of bootstrapping over tradi­tional parametric statistics is that it sometimes may bebetter to draw conclusions about the parameters of apopulation strictly from the sample at hand (e.g., a smallsample) than to make perhaps unrealistic assumptionsabout the population. Another advantage of bootstrap­ping is that it provides a useful alternative when formu­las for parametric statistics are not available. For example,the x% trimmed mean is a very useful measure of cen­tral tendency, when sample data come from a long-tailedpopulation (Huber, 1981). The robust estimator is ob­tained by "trimming" the data to exclude values far fromothers to produce a smaller measurement error. In thepast, however, there were no analytic parametric formu­las for computing the standard error of this statistic.Thus, the accuracy of the estimation could not be deter­mined, and the application of the statistic was limited.By using bootstrapping, the problem can be solved eas­ily. Bootstrap sampling distributions were generated forthe x% trimmed means (i.e., 20% trimmed mean in thisstudy) and the standard deviations of the distributionswere used as the estimated standard error of the statis­tics (see Table 2).

Bootstrapping has been applied to a variety offields:medicine (Koehler & McGovern, 1990), psychophysiol­ogy (Wasserman & Bockenholt, 1989), social science(Hand, 1990), and education (Maguire, 1986), as wellas a variety of statistics, such as factor analysis (Borrello

50

& Thompson, 1989), repeated measures analysis ofvari­ance (Lunneborg & Tousignant, 1985), and covariancestructure analysis (Bollen & Stine, 1990). Bootstrappinghas great application potential in the field of physicaleducation (PE) and exercise science because many of thedata distributions are skewed (e.g., pull-up data in fitnesstesting), and many researchers use small samples in theirstudies. For more information about bootstrapping, seethe introductory texts by Efron and Tibshirani (1993)and Mooney and Duval (1993), and a more technicalreference by Efron (1982).

Limitations of Bootstrapping

No statistical procedure always yields the correctanswer. This is true also for bootstrapping, especially ifthe resampling scheme does not parallel the structureof the actual sampling mechanism. Recall that abootstrapping sampling distribution is derived from one"original" sample from the population. If this originalsample is not a representative sample of the population,bootstrapping may give misleading results. Efron andTibshirani (1993) also reported that bootstrapping mayfail when the population distribution is extremelyskewed. Furthermore, a biased random number genera­tor could lead to the failure of bootstrapping; therefore,every effort should be made to control and monitor thefunction of the random number generator (see Ripley,1987, for related tests).

An Example

To demonstrate that bootstrapping is a useful alter­native and complement for parametric statistics, the au­thor illustrates how bootstrapping can assist indeveloping a confidence interval and making a bootstrapstatistical inference when a parametric analytic formulais not appropriate. An existing data set served as thepopulation being studied to compare bootstrap estimateswith the "true parameters."

Correlation is one of the most commonly used sta­tistical procedures in the field ofPE and exercise science.To validate a field test, for example, the scores from afield test often are correlated with the scores from a cri­terion measure. In reality, correlations vary across stud­ies, even when the same measures are applied, becauseofsampling error. Statisticians, therefore, have developedindexes, such as standard error and confidence intervals,to determine the accuracy of sample statistics. Followingis an example of how bootstrapping can help develop aconfidence interval ofa population correlation.

An existing body composition data set was used (Jack­son & Pollock, 1985), in which the percentage of bodyfat of 678 participants was determined by hydrostatic, orunderwater, weighing. A number ofanthropometric vari­ables, such as height, weight, and skinfolds, were also

ROES: March 1997

Dow

nloa

ded

by [

Uni

vers

ity o

f K

iel]

at 2

1:21

24

Oct

ober

201

4

measured. The data for the female participants (N= 279)were employed as the example population, and the bodymass index [BMI = Weight (kg)/Height2 (m)] was usedas the field test to predict participants' body fat. Thescores of BMI and percentage of body fat are plotted inFigure 3 (top). To validate the BMI, the author computedits correlation (i.e., Pearson's product-moment correla­tion coefficient, p) with percentage of body fat, the cri­terion measure; and p = 0.707. In real life, however,researchers often do not know the population parameterbut try to make an inference from a sample statistic. Togenerate a sample statistic, a small sample (n = 15 pairs)was drawn (see Table 3) and plotted in Figure 3 (bot­tom). The correlation coefficient of this sample (r) was0.733, which was the point estimate of the population pa­rameter (p). It provided a reasonably accurate estimateof the population parameter. Itwas known, however, that

Table 3.Sample, Fisher's ztransformation, and confidence intervals

Sample

Zhu

if another sample of the same size were drawn randomlyfrom the population, the new rmight be different fromthe previous one, due to sampling error. Therefore, itwas necessary to develop an interval estimate of the popu­lation parameter. The commonly used 95% confidenceinterval was employed as the interval estimate.

Because the author had the "population" data, a"real" sampling distribution of the correlation coeffi­cients could be developed according to the theoreticalapproach illustrated in Figure 1. Based on this samplingdistribution, a "true" 95% confidence interval could bedeveloped to compare with the intervals developed byother approaches. The author randomly drew another1,999 samples (each with n = 15 pairs) with replacement,computed correlation coefficients for each sample, andplotted them (N = 2,000) in a histogram (see the top ofFigure 4). The sampling distribution, as expected, was

Pair 1 2 3 4 5 6 7 8BMI 19.11 20.03 18.79 19.74 25.24 19.63 20.46 22.51Body fat % 25.04 22.32 20.08 19.19 37.68 12.61 19.28 25.23

Pair 9 10 11 12 13 14 15BMI 20.08 21.18 20.64 24.09 25.62 20.74 20.42Bodyfat % 17.51 17.06 22.05 42.04 28.72 29.65 24.59

Sample statistics

r= 0.733

Parametric estimation of 95% confidence interval

SElri =(l-f)/(n _1)1/2 =(1 - 0.7332)/(15-1)1/2 =0.124

[r-l.96 x SElri s p s r+ 1.96 x SElrl) =[0.733 - 1.96 x 0.124 s Ps 0.733 +1.96 x 0.124)

Fisher's zTransformation of95% Confidence Interval

z, =1/2[/n(1 + rI-/n(1 - rH =1/2[/n(1 +0.733) -/n(1-0.733)) =0.935

SElz,) =l/(n - 3)1/2 =1/(15-3j1J2 =0.289

[(z, - 1.96 x SElz,) ~ p ~ z, +1.96 x SElz,)) =[0.935 - 1.96 x 0.289 ~ P~ 0.935 +1.96 x 0.289) =[0.369 ~ P~ 1.501)

Transformed back: tanh(z,) = rtanh(0.369) =0.353; tanh(1.501) =0.905

Summary of95% confidence intervals

Real samples ("true"):Parametric formula:Fisher's z transformation:Bootstrap percentile:

Note. BMI=body mass index.

ROES: March 1997

[0.335 s p s 0.887)[0.490 s Ps 0.976)[0.353 s P~ 0.905][0.385 ~ Ps 0.897)

51

Dow

nloa

ded

by [

Uni

vers

ity o

f K

iel]

at 2

1:21

24

Oct

ober

201

4

Zhu

Bootstrap &lmptes .!I: :.!I

11:!:1:11.

~: :i.1 :..:til. -I ,01

I ,I ,I ,, II ,• .I

,, ,, ,• ,, ,I,1,,,, ..I,

hi :n

. Jl: ,Real Samp!&s !!' - :-1

~: 11cf:f r-

,~: '='Illtil: :,.;, .01, •, ,

I,, I

• - 1, I, ,II II I, •I I, r- I,

II, ,I II ,

AII,:l

,

o·.43 ·.28 ·.12 .03 .17 .32 .47 .62 .77 .93

50

100

150

450

100

P[r-1.96 x SE(r) ::;; p::;; r+ 1.96 x SE(r)] = 0.95, (3)

300

ric analytical formula for the correlation coefficientis as follows:

350

400

o-.43 •.28 -.12 .03 .17 .32 .41 .62 .77 .93

300

where SE (r) is the standard error of r: SE (r) = (1 - -r) /(n - 1) 1/2. The formula, however, is used rarely in prac­tice, because when the population correlation (p) dif­fers from zero and the sample is not extremely large, thesampling distribution of correlation coefficients is usu­ally skewed. This is demonstrated by the "true" samplingdistribution in Figure 4. Therefore, the confidence in­terval based on this formula is often incorrect. To illus­trate, the author computed an interval estimate, usingFormula 3 with the first sample, and the resulting 95%

+

30

+ ...of.

o

oo

ao 0 a

00000

00

o

45

<40

35

30at~ 25

I :m

15

1C

5

C1S

4S

40

35

30:I;iii 25iL

I 20

15

10

5

015 25

Body. MassIndex

negatively skewed, M = 0.691, SD = 0.145. Because the"real" sample statistics from the population wereknown, the confidence interval could be computeddifferently (i.e., confidence interval was no longerbased on the standard error, as usual, but developeddirectly from the sample statistics. Because the a levelwas set at 0.05, the corresponding percentiles for lo­cating the lower and upper endpoints of the confi­dence interval should be the 2.5th (0.05/2) percentileand the 97.5th (1- 0.05/2). Therefore, for the 95% con­fidence interval, the author simply sorted all thesample correlation coefficients (18) and counted upfrom the smallest sorted value to the 50th smallestvalue, which was 0.335. Similarly, the author counteddown from the highest value to the 50th highest value,which was 0.887. Thus, the "true" 95% confidence in­terval for the population correlation was 0.335 ::;; p ::;;0.887 (see Figure 4).

In reality, however, the population data did notexist for drawing many samples. Therefore, the con­fidence interval was developed from a single sampleusing the existing parametric formula. The paramet-

Correlation CoelficientFigure 3.The top panel is a scatterplot ofthe "population" data(N= 279); circles indicate the sample data that is plotted in thebottom panel.

Figure 4.Sampling distributions and the 95% confident intervals ofcorrelation coefficients based on realand bootstrap samples.

52 ROES: March 1997

Dow

nloa

ded

by [

Uni

vers

ity o

f K

iel]

at 2

1:21

24

Oct

ober

201

4

confidence interval was 0.490::; p s 0.976 (see Table 3).As expected, it was an inaccurate estimation when com­pared with the "true" confidence interval.

One alternative, as described earlier, is to apply atransformation, when assumptions of parametric statis­tics have been violated. For correlation coefficients, for­tunately, a transformation procedure, known as Fisher'sZ transformation, was developed. More specifically, thesample rwas first transformed to zr through Fisher's Z

transformation (see Table 3). The 95% confidence in­terval of zr' based on a standard error, was then devel­oped. The standard error of zr is a function of samplesize, SE(zr) = I/(n - 3)1/2. After converting zr back to r,the 95% confidence interval based upon the transfor­mation was developed (0.353::; Ps 0.905). By compar­ing this interval with the "true" confidence interval, itwas clear that the transformation was successful.

Transformation procedures, however, are not with­out their limitations. First, few work as well as Fisher's Z

transformation. Second, the transformations may be lim­ited by the assumptions on which they are based. Forexample, Fisher's Z transformation assumes that corre­lated variables follow a bivariate normal distribution.While the violation of this assumption is often robust,the estimation could be misleading when the populationcorrelation is high and the sampling distribution ofzrismarkedly skewed. Finally, a violation of the assumptionsrelated to the transformation may not be easy to detect.For example, it is difficult to detect the degree of theviolation of the "bivariate normal distribution" assump­tion, when little is known about the population.

Bootstrapping can help in developing a confidenceinterval free from the limitations of often unrealisticand unverifiable assumptions. Bootstrapping statisti­cal inference, as illustrated in Figure 1, is developedin a similar paradigm as the theoretical approach. Theonly difference is that in the theoretical approach, manysamples are redrawn from the population, while inbootstrapping, many samples are redrawn from a singlesample. The steps to developing a bootstrap confidenceinterval for rho (p) follow:

Step 1: Draw a Random Sample. To develop a bootstrapconfidence interval of a parameter (the populationcorrelation between BMI and percentage of body fat,in this case), a sample was drawn from the populationdescribed earlier (see Figure 3 and Table 3).

Step 2: Redraw Bootstrap Samples. Next, many boot­strap samples were redrawn, randomly and with replace­ment, from the sample to develop a bootstrap samplingdistribution. A total of 2,000 bootstrap samples (eachwith n = 15 pairs) were drawn. These procedures werecompleted by using BOJA bootstrap computer software(Boomsma, 1991).

Step 3: Compute and Sort Bootstrap Sample Statistics. Todevelop a bootstrap sampling distribution, bootstrapsample statistic, in this case, the correlation between the

ROES: March 1997

Zhu

BMI and percentage of body fat, was computed. For ex­ample, the correlation coefficient for the first bootstrapsample was 0.689 and 0.730 for the second bootstrapsample. Similar computations were repeated for all 2,000bootstrap samples. All correlation coefficients were thensorted, which consisted of the bootstrap sampling dis­tribution (see the bottom of Figure 4); M= 0.724, SD=0.132, which provided an accurate estimate of the "true"sampling distribution. Furthermore, the bootstrap sam­pling distribution, similar to the "true" sampling distri­bution, was negatively skewed.

Step 4: Determine the Confidence Interval. The confi­dence interval was computed using the percentilemethod. As with the "real" samples, all bootstrap corre­lation coefficients were sorted. Then, the authorcounted up from the smallest sorted value to the 50thsmallestvalue, which was 0.385, and counted down fromthe highest value to the 50th highest value, which was0.897. Thus, the 95% percentile bootstrap confidenceinterval of the population correlation was 0.385 ::; p ::;0.897. Bycomparing it with the "true" confidence inter­val, the bootstrapping also provided an accurate estima­tion (see Table 3 and Figure 4). Generally, the choiceof bootstrap confidence interval method depends onthe practical research situation a researcher faces. Formore information about these methods, refer toMooney and Duval (1995) for a general introductionand Efron (1982) and Efron and Tibshirani (1993) fortechnical details.

Step 5: Make Statistical Inference. Based on the boot­strap confidence interval developed by the percentilemethod, the author concluded, with 95% confidence,that the interval 0.385-0.897 covered the populationcorrelation (p) between participant' body mass indexesand their percentage of body fat.

One may argue that a similar accurate estimationwasachieved by Fisher's Z transformation. While it is truethat Fisher's Z transformation worked well in this case,bootstrapping demonstrated three advantages over con­ventional procedures in developing the confidence in­terval of p. First, bootstrapping, which is free fromdistribution assumptions, provided a more accurate es­timation of the "true" confidence interval than the para­metric approach, whose assumption has been violated.Second, bootstrapping provided a similar estimation ofthe "true" confidence interval as the well developedFisher's Z transformation, which sometimes may be lim­ited by its own assumption. Indeed, there is no need toapply Fisher's z transformation when bootstrapping isemployed, because bootstrapping automatically accom­plishes much ofwhat Fisher's z transformation does (seeStine, 1990, for a more detailed explanation). Further­more, few well developed transformations, such asFisher's z, are available for practical purposes. Finally, itis much simpler and faster to construct a bootstrap con­fidence interval using bootstrapping methods than by

53

Dow

nloa

ded

by [

Uni

vers

ity o

f K

iel]

at 2

1:21

24

Oct

ober

201

4

Zhu

using conventional methods. The computations ofFisher's z transformation illustrated in Table 3, althoughnot too complex, can confuse the user. In contrast, byusing a 486 personal computer and BOJA software, theauthor was able to construct the bootstrapping confi­dence intervals and compute other useful information(e.g., descriptive statistics) in less than 1 min. It is ex­pected that bootstrapping applications will become evenmore convenient as computer speed increases and com­monly used statistical software packages integratebootstrapping into their analytical routines.

However, bootstrapping may not work all the timeor under all conditions. For example, Rasmussen(l987) found that bootstrapping resulted in overlyrestricted confidence intervals and overly liberal TypeI error rates of correlation coefficients. Others (Efron,1988; Strube, 1988), however, reported that bootstrapintervals of the correlation coefficient performedquite well. At any rate, bootstrapping statistical infer­ence is a more appropriate choice when an analyticformula is not available, when the assumption of aparametric analytic formula is violated, or when theassumption of a transformation is suspect.

Conclusion

The development of bootstrapping, a major com­puter-intensive statistics, has provided not only a non­parametric" approach to analyzing data for whichparametric statistics are not appropriate but also a newway to explore statistics that traditionally cannot be ap­plied due to a violation of statistical assumptions or alack of analytic formulas. The key idea, computations,and advantages of bootstrapping were illustrated in thisstudy using a set of real data from the field. An exampleof how to apply bootstrapping to statistical inference wasalso provided. The major advantage of bootstrappingover traditional statistics is that sometimes it may bebetter to draw conclusions about the parameters of apopulation strictly from the sample at hand rather thanmaking perhaps unrealistic assumptions about the popu­lation. Finally, bootstrap-ping should not be consideredpreferable to parametric statistics but rather as an alter­native, or supplement, when parametric statistics areneither appropriate nor available.

References

Bollen, K A, & Stine, R. A (1990). Direct and indirect effects:Classical and bootstrap estimates of variability. In C. C.Clogg (Ed.), SociologicalMethodology 1990 (pp. 115-140).Oxford, England: Blackwell.

54

Boomsma, A (1991). BOJA: A programfor bootstrapandjackknifeanalysis [Computer program]. Groningen, The Nether­lands: iec ProGAMMA.

Borrello, G. M., & Thompson, B. (1989). A replication boot­strap analysis of the structure underlying perceptions ofstereotypic love. TheJournal ofGeneralPsychology, 116, 317­327.

Budescu, D. V., & Appelbaum, M. I. (1981). Variance stabiliz­ing transformations and the power of the F test. JournalofEducational Statistics, 6,55-74.

Diaconis, P., & Efron, B. (1983). Computer-intensive methodsin statistics. Scientific American, 248(5), 116-130.

DiCiccio, T.]., & Romano,]. P. (1989). The automatic percen­tile method: Accurate confidence limits in parametricmodels. CanadianJournalofStatistics, 17,155-169.

Efron, B. (1979). Bootstrap methods: Another look at thejack­knife. The Annals ofStatistics, 7, 1-26.

Efron, B. (1982). Thejackknife, the bootstrapand otherresamplingplans. Philadelphia: Society for Industrial and AppliedMathematics.

Efron, B. (1987). Better bootstrap confidence intervals.Journal of the American Statistical Association, 82, 171­185.

Efron, B. (1988). Bootstrap confidence intervals: Good orbad? PsychologicalBulletin, 104, 293-296.

Efron, B. (Speaker). (1995). Computers, bootstraps, and statis­tics (Cassette Recording No. RA5-31.03). Washington, DC:American Educational Research Association.

Efron, B., & Tibshirani, R. (1986). Bootstrap methods for stan­dard errors, confidence intervals, and other measures ofstatistical accuracy. Statistical Science, 1,54-77.

Efron, B., & Tibshirani, R. (1993). An introduction to the boot­strap. New York: Chapman & Hall.

Hall, P., & Wilson, S. R. (1991). Two guidelines for bootstraphypothesis testing. Biometrics, 47,757-762.

Hand, M. L. (1990). A resampling analysis of federal familyassistance program quality control data: An applicationof the Bootstrap. Evaluation Review, 14,391-410.

Hays, W. (1981). Statistics (3rd ed). New York: CBS CollegePublishing.

Hill, M., & Dixon, W.J. (1982). Robustness in real life: A studyof clinical laboratory data. Biometrics, 38, 377-396.

Hjorth, U. (1994). Computer intensive statistical methods: Valida­tion model selection and bootstrap. New York: Chapman &Hall.

Huber, P. (1981). Robust statistics.New York:John Wiley & Sons,Inc.

Jackson, AS., & Pollock, M. L. (1985). Practical assessmentofbody composition. The Physician and Sportsmedicine, 13,76-90.

Kirk, R. E. (1982). Experimental designfor the behavioral sciences(2nd ed.). Belmont, CA: Brooks/Cole.

Koehler, K]., & McGovern, P. G. (1990). An application ofthe LEP survival model to smoking cessation data. Statis­tics in Medicine, 9,409-421.

Lunneborg, C. E. (1987). Bootstrap applications for the be­havioral sciences. Educational and Psychological Measure­ment, 47,627-629.

Lunneborg, C. E., & Tousignant,]. P. (1985). Efron's boot­strap with application to the repeated measures design.Multivariate Behavioral Research, 20, 161-178.

ROES: March 1997

Dow

nloa

ded

by [

Uni

vers

ity o

f K

iel]

at 2

1:21

24

Oct

ober

201

4

Maguire, T. O. (1986). Applications of new directions in sta­tistics to educational research. The Alberta Journal ofEducational Research, 2, 154-171.

Miller, R. G. (1974). Thejackknife: A review. Biometrika, 61,1-15.

Mooney, C. Z., & Duval, R. D. (1993). Bootstrapping; A non­parametric approach to statistical inference. Newbury Park,CA: Sage.

Noreen, E. W. (1989). Computer-intensive methods for testinghypotheses. New York: John Wiley & Sons, Inc.

Quenouille, M. H. (1949). Approximate tests of correlationin time-series. Journal of the Royal Statistical Society, 11,(Series B) 68-84.

Rasmussen,]. L. (1987). Estimating correlation coefficients:Bootstrap and parametric approaches. PsychologicalBul­letin, 101, 136-139.

Ripley, B. D. (1987). Stochastic simulation. New York: JohnWiley & Sons, Inc.

Rohatgi, V. K. (1984). Statistical inference. New York: JohnWiley & Sons, Inc.

Ross,]. G., & Gilbert, G. G. (1985). The national childrenand youth fitness study: A summary of findings. Jour­nal ofPhysical Education, Recreation, and Dance, 56, 45­50.

Stine, R. (1990). An introduction to bootstrap methods: Ex­amples and ideas. Sociological Methods and Research, 18,243-291.

Strube, M.]. (1988). Bootstrap Type I error rates for cor­relation coefficient: An examination of alternate pro­cedures. Psychological Bulletin, 104, 290-292.

Tukey, ]. W. (1958). Bias and confidence in not quite largesamples (Abstract). The Annals ofMathematical Statistics,29,614.

Wasserman, S., & Bockenholt, U. (1989). Bootstrapping:Applications to psychophysiology. Psychophysiology, 26,208-221.

ROES: March 1997

Zhu

Notes

1. The population is usually unknown in studies. Forthe purpose of illustration, this national sample is de­fined as "population" so that bootstrap estimates couldbe compared with known parameters.2. Throughout, the author used e to denote a true pa­rameter of a population and eto denote an estimate ofe based on sample data.3. Bootstrapping, although less accurate than permu­tation tests, can also be used in hypothesis testing, and,often, it is more widely applicable than permutationtests. For a more thorough treatment on this topic, re­fer to Efron and Tibshirani (1993).4. S-Plus, developed originally at AT&T's Bell Labora­tories, is a data analysis tool published by Statistical Sci­ences, Inc.5. An idea of "parametric bootstrap" has also been pro­posed; refer to Efron and Tibshirani (Chapter 21,1993)for more detail.

Author's Notes

The author thanks Andrew S.Jackson of the Universityof Houston for allowing the use of his data and RobertW. Schutz of the University of British Columbia andMarilyn A. Looney of Northern Illinois University fortheir helpful comments and suggestions on the manu­script. Please direct all correspondence regarding thisarticle to Weimo Zhu, Division of Health, Physical Edu­cation, and Recreation, Wayne State University, 257Matthaei Building, Detroit, MI 48202.

E-mail: userbr8f@mts.cc.wayne.edu

55

Dow

nloa

ded

by [

Uni

vers

ity o

f K

iel]

at 2

1:21

24

Oct

ober

201

4