Introduction to the Bootstrap
Machelle D. Wilson
Outline
Why the Bootstrap? Limitations of traditional statistics
How Does it Work? The Empirical Distribution Function and the
Plug-in Principle Accuracy of an estimate: Bootstrap standard
error and confidence intervals Examples How Good is the Bootstrap?
Limitations of Traditional Statistics: Problems with distributional assumptionsOften data can not safely be assumed
to be from an identifiable distribution.Sometimes the distribution of the
statistic is mathematically intractable, even assuming that distributional assumptions can be made.
Hence, often the bootstrap provides a superior alternative to parametric statistics.
An example data set
80 100 120 140 160 180
050
100
150
200
250
1000 Bootstrapped Means
Mean conc. and Dose rate fixedmean dose
Red Lines=BS CI
Black Lines=Normal CI
An Example Data Set
50 100 150 200 250 300 350
050
100
150
200
250
300
1000 Bootstrapped Means
Mean Conc and Dose Rate Randommean dose
Red Lines=BS CIBlack Lines=Normal CI
Statistics in the Computer Age
Efron and Tibshirani, 1991 in Science: “Most of our familiar statistical methods, such as hypothesis testing, linear regression, analysis of variance, and maximum likelihood estimation, were designed to be implemented on mechanical calculators. Modern electronic computation has encouraged a host of new statistical methods that require fewer distributional assumptions than their predecessors and can be applied to more complicated statistical estimators…without the usual concerns for mathematical tractability.”
The Bootstrap Solution
With the advent of cheap, high power computing, it has become relatively easy to use resampling techniques, such as the bootstrap, to estimate the distribution of sample statistics empirically rather than making distributional assumptions.
The bootstrap resamples the data with equal probability and with replacement and calculates the statistic of interest at each resampling. The resulting histogram, mean, quantiles and variance of the bootstrapped statistics provide an estimate of its distribution.
Example
Take the data set 1,2,3. There are 10 possible resamplings, where re-orderings are considered the same sampling.
1,2,3 1,1,2
1,1,3 2,2,1
2,2,3 3,3,1
3,3,2 1,1,1
2,2,2 3,3,3
1 1.2 1.4 1.6 1.8 2 2.2 2.4 2.6 2.8 30
5
10
15
20
25
30
The Bootstrap Solution
In general, the number of bootstrap samples, Cn, is
Table of possible distinct bootstrap re-samplings by sample size.
2 1.
1n
nC
n
n 5 10 12 15 20 25 30
Cn 126
92,378
1.35x104
7.76x105
6.89x1010
6.32x1013
5.91x1016
The Empirical Distribution FunctionHaving observed a random sample of
size n from a probability distribution F,
the empirical distribution function (edf), assigns to a set A in the sample space of x its empirical probability
1 2, ,... nF x x x
ˆ ,F
ˆ ˆ # /iF A P A x A n
Example
A random sample of 100 throws of a die yields 13 ones, 19 twos, 10 threes, 17 fours, 14 fives, and 27 sixes. Hence the edf is
ˆ (1) 0.13
ˆ (2) 0.19
ˆ (3) 0.10
F
F
F
ˆ (4) 0.17
ˆ (5) 0.14
ˆ (6) 0.27
F
F
F
The Plug-in Principle
It can be shown that is a sufficient statistic for F.
That is, all the information about F contained in x is also contained in .
The plug-in principle estimates
by
F̂
F̂
( )T F
ˆ ˆ( )T F
The Plug-in Principle
If the only information about F comes from the sample x, then is a minimum variance unbiased estimator of .
The bootstrap is drawing B samples from the empirical distribution to estimate B statistics of interest,
Hence, the bootstrap is both sampling from an edf (of the original sample) and generating an edf (of the statistic).
ˆ ˆ( )T F
*ˆ .
Graphical Representation of the Bootstrap
x={x1,x2,…,xn}
x*1 x*2 x*3 …. …. … x*B
T(x*1) T(x*2) T(x*3) … … T(x*B)
2
1
[ ( ) ]ˆ ( ( )) 1
Bb
b
T x tse T x B
*
1
1ˆ ( )
Bb
b
t T x T xB
Bootstrap Standard Error and Confidence intervals.
The bootstrap estimate of the mean is just the empirical average of the statistic over all bootstrap samples.
The bootstrap estimate of standard error is just the empirical standard deviation of the bootstrap statistic over all bootstrap samples.
Bootstrap Confidence Intervals
The percentile interval: the bootstrap confidence interval for any statistic is simply the /2 and 1-/2 quantiles.
For example, if B=1000, then to construct the BS confidence interval we rank the statistics and take the 25th and the 975th values.
There are other BS CIs but this is the easiest and makes the fewest assumptions.
Example: Bootstrap of the MedianGo to Splus.
How Good is the Bootstrap?
The bootstrap, in most cases is as good as the empirical distribution function.
The bootstrap is not optimal when there is good information about F that did not come from the data–i.e. prior information or strong, valid distributional assumptions.
The bootstrap does not work well for extreme values and needs some what difficult modifications for autocorrelated data such as times series.
When all our information comes from the sample itself, we can not do better than the bootstrap.