Running head: ESTIMATING THE VARIANCE · Estimating the Variance 2 Abstract Problem: To simplify the computation of the variance in before-after studies, it is generally assumed that

Estimating the Variance

1

Running head: ESTIMATING THE VARIANCE

Estimating the Variance in Before-After Studies

Zhirui Ye1

Western Transportation Institute

Montana State University

P O Box 174250

Bozeman, MT 59717

Phone (406) 994-7909

e-mail: [email protected]

Dominique Lord

Zachry Department of Civil Engineering

Texas A&M University

3136 TAMU

College Station, TX 77843

Tel (979) 458-3949

Fax (979) 845-6481

e-mail: [email protected]

March 11, 2009 1Corresponding author


2

Abstract Problem: To simplify the computation of the variance in before-after studies, it is generally

assumed that the observed crash data for each entity (or observation) are Poisson distributed.

Given the characteristics of this distribution, the observed value ( ix ) for each entity is implicitly

made equal to its variance. However, the variance should be estimated using the conditional

properties of this observed value (defined as a random variable), i.e. ( )|i if x μ , since the mean

of the observed value is in fact unknown. Method: parametric and non-parametric bootstrap

methods were investigated to evaluate the conditional assumption using simulated and observed

data. Results: The results of this study show that observed data should not be used as a substitute

for the variance, even if the entities are assumed to be Poisson distributed. Consequently, the

estimated variance for the parameters under study in traditional before-after studies is likely to be

underestimated. Conclusions: the proposed methods offer more accurate approaches for

estimating the variance in before-after studies.

Keywords: Before-after study; Variance estimation; Bootstrap; Resampling; Non-parametric


3

1. Introduction

The before-after study is a commonly used method for measuring the safety effects of a single

treatment or a combination of treatments in highway safety (Hauer, 1997). Short of a controlled

and full randomized study design, this type of study is deemed superior to cross-sectional studies

since many attributes linked to the converted sites where the treatment (or change) was

implemented remain unchanged. Although not perfect, the before-after study approach offers a

better control for estimating the effects of a treatment. In fact, as the name suggests, it implies

that a change actually occurred between the “before” and “after” conditions (Hauer, 2005a).

Combined with the empirical Bayes (EB) technique, the before-after study can also minimize the

bias caused by the regression-to-the-mean (RTM) commonly found in crash data analyses

(Persaud et al., 2001 & 2003). Despite their large popularity, it should be mentioned that not

everyone agrees about their superiority over cross-sectional studies (Tarko et al., 1998; Noland,

2003).

Before-after studies can be grouped into three types: the simple (naïve) before-after study,

the before-after study with control groups, and the before-after study using the EB technique

(also using a control group).The selection of the study type is usually governed by the

availability of the data, such as crashes and traffic flow, and whether the transportation safety

analyst has access to entities that are part of the reference group. The selection can also be

influenced by the amount of available data (or sample size).

As described by Hauer (1997), the traditional before-after study (no matter which type is

used) can be accomplished using two tasks. The first task consists of predicting the expected

number ( π̂ ) (in this paper, we will work with the estimated value; hence, π̂ is an estimate of π )

of target crashes for a specific entity (i.e., intersection, segment, etc.) or series of entities in the


4

“after” period had the safety treatment not been implemented. The second task consists of

estimating the number of target crashes ( λ̂ ) for the specific entity in the “after” period. Here, the

term “after” means the time period after the implementation of a treatment; correspondingly, the

term “before” refers to the time before the implementation of this treatment. In most practical

cases, either π̂ or λ̂ can be applied to a composite series of entities where a similar treatment

was implement at each entity.

Hauer (1997) proposed a four-step process for estimating the safety effects of a treatment.

The process is described as follows:

Step 1: For n ..., ,2 ,1=j , estimate ( )jλ and ( )jπ . Then, compute the summation of the

estimated and predicted values, such that ˆ ˆ( )jλ λ=∑ and ˆ ˆ( )jπ π=∑ .

Step 2: For n ..., ,2 ,1=j , estimate )}(ˆ{ jVar λ and )}(ˆ{ jVar π . For each single entity, it is

assumed that observed data (e.g., annual crash counts over a long timeframe) are Poisson

distributed and ˆ( )jλ can be approximated by the observed value in the before period. On the

other hand, the calculation of )}(ˆ{ jVar π will depend on the statistical methods adopted for the

study (e.g., observed data in naïve studies, method of moments, regression models, EB

technique). Assuming that crash data in the before and after periods are mutually independent,

then ∑= )}(ˆ{}ˆ{ jVarVar λλ and ∑= )}(ˆ{}ˆ{ jVarVar ππ .

Step 3: Estimate the parameters δ and θ , where λπδ ˆˆˆ −= (again, referring to estimated

values) is defined as the reduction (or increase) in the number of target crashes between the

predicted and estimated values, and πλθ ˆ/ˆˆ = is the ratio between these two values. The term θ

has also been referred to in the literature as the index of effectiveness (Persaud et al., 2001).

Hauer (1997) suggests that when less than 500 crashes are used in the before-after study, θ


5

should be corrected to remove the bias caused by the small sample size using the following

adjustment factor ]ˆ/}ˆ{1/[1 2ππVar+ .

Step 4: Estimate the variances }ˆ{δVar and }ˆ{θVar . These two variances are calculated using

the following equations (note: }ˆ{θVar is also adjusted for the small sample size) below:

}ˆ{}ˆ{}ˆ{ πλδ VarVarVar += (1)

22

222

)]ˆ/}ˆ{(1[)]ˆ/}ˆ{()ˆ/}ˆ{[(ˆ

}ˆ{ππ

ππλλθθVar

VarVarVar+

+= (2)

The four-step process provides a simple way for conducting before-after studies. One

important assumption with this process is related to the computation of the variance ˆ( )Var λ (or

ˆ{ }Var π ). As described above, observed crash data are assumed to be Poisson distributed for each

entity and the observed data are directly used in the analysis. However, as noted by Hauer

(1997), the variance ˆ( )Var λ is in fact unknown. The properties of the Poisson distribution are in

essence used to simplify the computation of the variance. In this case, the observed crash counts,

used here as random variables, are used as a substitute for estimating the variance for each entity

(i.e., the observation is assumed to be equal to its mean).1 Given the fact the mean of the

1 To examine whether this assumption in before-after studies is reasonable, one can look into a single entity. If the actual expected number of crash counts ( )( jη ) for the jth entity in either the before or the after period, then it can

be shown that ˆ ˆ ˆ{ ( )} { ( )} ( )Var j Var j jη η η η= =∑ ∑ . In practice, )( jη is approximated using observed

data )( jx (a random variable) and the true mean is therefore not known with certainty. By using the probability mass function (PMF) of the Poisson distribution, it is straightforward to compute the probability that

)()()}(ˆ{ jxjjVar ==ηη , assuming that )( jx represents the crash count over one year time period (or other very short time periods). It is obvious that the probability for the observed count (X) to equal the mean decreases as the mean )( jη increases. For example, the probability is about 40 percent when 1)( =jη , while it decreases to 10

percent when 15)( =jη . This entails that )()}(ˆ{ jxjVar =η may not be reasonable, since the count has a large probability not being equal to the “true” mean of an entity (if known). Thus, it is safe to assume that )( jx cannot be a good approximation of ( )jη .


6

observed value is unknown, the variance should be estimated using the conditional properties of

the observed value, i.e. ( )|i if x μ (e.g., see Cook and Wei, 2001; Diggle et al., 2002).

Consequently, there is a need to evaluate how these conditional properties affect the estimation

of the variance in a context of a before-after study.

The objectives of this paper are to evaluate whether or not the assumption that crash data

should be used a direct substitute to the variance is valid, even when one assumes the data are

Poisson distributed for each entity, and if not, to examine how this may affect the estimation of

the variance for calculating the inferences associated with the parameters used to estimate the

safety effects in a before-after study. To accomplish the objectives of this study, parametric and

non-parametric bootstrap resampling methods are investigated to evaluate this assumption. The

bootstrap method is first applied to data simulated using a Poisson distribution and a Negative

Binomial (or Poisson-gamma) distribution, and a mixture of these two distributions to evaluate

its applicability in this research. Then, both methods are applied to two datasets of observed

before and after crash data taken from the literature. The proposed methods are used to estimate

λ̂ , π̂ , ˆ{ }Var λ and ˆ{ }Var π and the output is compared with the traditional before-after method

to compute these values.

The rest of this paper is divided into six sections. The first section presents the parametric

method for estimating the variance of conditional random variables. The second section presents

the characteristics of the bootstrap method used in this study. The third section covers the

evaluation of the bootstrap method using simulated data. The fourth section presents the

application of the methods to observe before and after data taken from the literature. The fifth

section describes important discussion points associated with before-after studies and offers

avenues for further work. The last section summarizes the key findings of this study.


7

2. Parametric Method

Since the mean of an entity is unknown, the analysis of the random variable must be carried

out using the conditional properties of this variable with respect to the mean, i.e. ( )f x η , where

{ }(1), (2), , ( )jη η η η= L (Cook and Wei, 2001; Agresti, 2002; Bolstad, 2004). Furthermore, the

conditional property entails that the values can be approximated using any suitable distribution.

Researchers, who have conducted before-after studies (non-randomized trials) using the same

dataset in the before and after periods, have analyzed the data using the conditional properties

described above (note: in many cases, the mean of the observation is modeled as a random-effect

variable). Examples of such observational studies where the mean of the Poisson distribution was

modeled as a random-effect variable can be found in medicine (Cook and Wei, 2001),

epidemiology (Laird and Ware, 1982; Diggle et al., 2002), and animal science (Schaik et al.,

1999). Very recently, researchers in highway safety have also started using this conditional

property for before-after studies (Persaud et al., 2009; Park et al., 2009). Depending upon

assumptions, the random-effect variable has been modeled using different marginal distributions.

For example, Diggle et al. (2002) have proposed the Gaussian distribution for modeling the mean

of the Poisson model. Because of the properties associated with the Poisson-gamma distribution

(i.e., the closed form of the conjugate distribution), other researchers have proposed to model the

mean using the gamma distribution (Cook and Wei, 2001; Diggle et al., 2002; Persaud et al.,

2009; Park et al., 2009). As discussed by Lord et al. (2005), it is important to point out that the

Poisson-gamma distribution (as well as other mixed-Poisson distributions, such as the Poisson-

lognormal) is used to approximate the true characteristics of motor vehicle crash process. It

should be pointed out that by allowing the mean to follow a given distribution, the variance

estimated will not be underestimated when a regression model is used in a before-after study.


8

Getting back to the primary objective of this analysis, if the means of the Poisson

distributions for entities 1, 2,..., j are assumed to follow a gamma distribution

( )/,(~)}(),...,1({ φμφηηη Gammaj= ), it can be shown that the marginal distribution becomes

the conjugate Poisson-gamma distribution, where φ is defined as the inverse dispersion

parameter of the Poisson-gamma distribution. The mean and variance of η are μ and 2 /μ φ ,

respectively.

Using the theorem proposed by Casella and Berger (1990), referred to as Conditional

Variance Identity (CVI), it is possible to estimate the variance for a series of observed values,

when each value is conditional upon the mean. This theorem states that “for any two random

variables X and Y, { } [ ( )] [ ( )]Var Y E Var Y X Var E Y X= + , provided that the expectations exist.”

For the curious reader, Agresti (2002) provides a very good discussion about the application of

the CVI properties to Poisson random variables. His discussion in fact supports this work. This

author states that the CVI needs to be used to estimate the variance of random variables because

μ varies (i.e., unknown) due to unmeasured factors.

Using η=X and ( )Y jη=∑ , the variance ( { }Var Y ) can be derived as

follows: 2{ } /Var Y n nμ μ φ= + , where μ can be approximated by the mean value of entities

1

1ˆ ( )n

jn

μ η= ∑ , and the inverse dispersion parameter φ can be estimated from the sample of

count data. Several estimators have been proposed to estimate the inverse dispersion parameter.

They include the Method of Moments Estimate (MME) (Anscombe, 1949), the Maximum

Likelihood Estimate (MLE) (Fisher, 1941), the Maximum Quasi-Likelihood Estimate (MQLE)

(Clark and Perry, 1989), the multistage method (Willson et al., 1984), and the Bootstrapped

Maximum Likelihood Estimate (BMLE) (Zhang et al., 2007).


9

3. Non-Parametric Bootstrap Resampling Method

In addition to the parametric method, a non-parametric method of bootstrap resampling was

also used for estimating variance in this study (Efron and Tibshirani, 1993). The bootstrap

method is distribution-free and makes no assumption about the distribution of the observed data.

Furthermore, the bootstrap method is usually not limited by the number of observations needed

to get a good estimate of the mean and variance of the data, although it is suggested to use a

sample size larger than five observations (Zoubir and Iskander, 2004). As discussed below,

smaller sample size tends to underestimate the true variance of the distribution (Rich, 2001). This

characteristic is unfortunately very common in highway safety (Lord, 2006). Nonetheless, it

should noted that the samples should have a good representation of the true distribution F

(although we may not know its characteristics), since one cannot make good inferences based on

unreliable data. Because of its useful properties, the non-parametric bootstrap resampling method

has been used extensively in many fields of research, including geophysics (Tauxe et al., 1991;

Kawano and Higuchi, 1995), biomedical engineering (Haynor and Woods, 1989), and image

processing (Hall, 1989; Saradhi and Murty, 2001) among others.

Another advantage of the bootstrap method is related to the relative easiness for computing

the values associated with the mean and variance. This computation can be accomplished using

commercially available statistical software programs. For example, the software statistical

program R (R Development Core Team, 2007), the one used in this study, provides a subroutine

for performing bootstrap resampling. To extract a resample, one can simply use the following

function in R.

),( Treplacexsample =

where x is the observed data, and T means TRUE for replacement.


10

4. Simulation Analysis for the Bootstrap Method

Simulation runs were performed in this part of this study to evaluate and validate the

applicability of the non-parametric bootstrap method described above. Three series of

simulations were analyzed. For the first two series, count data were generated using a Poisson or

NB distribution for different sample sizes and distribution parameters, respectively. For the third

series, count data were generated using a joint mixture of Poisson and NB distributions. The

mean (λ ) for the Poisson and NB distributions were equal, but the variance for each observation

followed either one of the two distributions. Different proportions assigned to each distribution

were evaluated. The three series were used to compare and validate the theoretical variances with

the values obtained from the bootstrap output.

5.1 Simulation Framework

The data were simulated using a Monte Carlo simulation approach. All simulation runs were

executed in R (R Development Core Team, 2007). The steps to simulate the data for the first two

series were as follows:

1) Generate n i.i.d. variables from a Poisson distribution with mean value λ , or from a NB

distribution with a mean value λ and an inverse dispersion parameter φ :

)(~},...,,{ 21 λχ PoissonXXX n= or 1 2{ , ,..., } ~ ( , )nX X X NBχ λ φ= . Thus, n here

refers to the sample size.

2) For the Poisson simulations, calculate the theoretical sum of variables ( )S nχ λ= , and

the theoretical variance of the sum )(χS is equal to this sum. For the NB simulations,

calculate the sum of variables ( )S nχ λ= , then the variance of the sum )(χS can be

calculated using 2{ ( )} /Var S n nχ λ λ φ= + .


11

3) Draw a sample from χ with replacement to obtain },...,{ **1

*nj XX=χ , then calculate the

sum of the resample ∑=

=n

iij XS

1

** )(χ ; repeat this resampling step B times to obtain

)(),...,( **1 BSS χχ .

4) Calculate the variance of )(),...,( **1 BSS χχ , that is, the bootstrap estimate of variance

}{ *SVar .

5) For the Poisson simulations, calculate the ratio )(/}{ * χSSVarr = . For the NB

simulations, calculate the ratio )}({/}{ * χSVarSVarr = . The closer the ratio r to 1, the

better the bootstrap estimate of the variance.

6) Repeat steps 1 to 5 for m times. Calculate the mean and variance of sample sum,

}{ *SVar , and r . Thus, the expected value and standard deviation of r are used for the

evaluation of the bootstrap method.

The initial conditions, for the first two series of simulation runs, were the following:

The sample size ( n ): 20, 50, and 100;

The sample mean of the Poisson distribution (λ ): 1, 5, 10, and 20;

The sample mean and inverse dispersion parameter of the NB distribution ( , )λ φ : (1, 2),

(5, 1), (5, 2), and (20, 2);

The number of resamples ( B ): 1000 times;

The number of iterations ( m ): 100 times.

For the third series, the same steps described above were used, but the proportion of count

data generated by the Poisson and NB distributions were as follows ( 100n = ):

Scenario 1: Poisson = 50% ( 20λ= ), NB = 50% ( 20λ= , 2.0φ= )


12



As above, 100 iterations ( m ) were used.

5.2 Simulation Results

The simulation results for the first series (Poisson distribution) are shown in Table 1 for the

sample sizes equal to 20, 50 and 100, respectively. From this table, the mean values of the

variance estimated by the non-parametric bootstrap method are very similar to the theoretical

values. Recall that the mean values for the sample sum were estimated using 100 iterations.

Overall, the bootstrap method performed very well for estimating the variance. As expected, with

a larger sample size, the bootstrap output provides better estimates. In fact, the variances of the

ratio are decreasing with an increase in sample size. Similarly, with a larger sample size, the

number of observations will provide a better estimate of the true distribution (see Lehmann,

1983, p. 114 for a discussion about the effects of sample size on the estimate of the Poisson

distribution).


13

Table 1 Results for the Poisson simulation

Sample Size

Mean Value λ

Theoretical Sum (or

Variance))(χS

Resampling Variance

))(( * χSVar

Ratio r a

mean variance MSE b

20

0.5 10 9.61 0.96 0.15 0.15

1 20 18.05 0.90 0.10 0.10

5 100 91.92 0.92 0.10 0.10

10 200 187.86 0.94 0.09 0.09

20 400 372.32 0.94 0.08 0.08

50

0.5 25 24.79 0.99 0.08 0.08

1 50 48.8 0.98 0.06 0.06

5 250 247.33 0.99 0.04 0.04

10 500 496.55 0.99 0.04 0.04

20 1000 980.34 0.98 0.03 0.03

100

0.5 50 50.10 1.00 0.04 0.04

1 100 100.23 1.00 0.03 0.03

5 500 502.35 1.00 0.03 0.03

10 1000 1011.8 1.01 0.03 0.03

20 2000 2017.4 1.00 0.02 0.02 Note:

a The ratio of bootstrap variance to theoretical variance ( )(/))(( * χχ SSVar ). 100 r values are generated in each parameter setting (or each row) from the simulation. The mean and variance are calculated based on these 100 values. b MSE: Mean Square Error. MSE = (Bias)2 + Variance.

The simulation results for the second series (NB distribution) are summarized in Table 2. As

described above, the NB simulations are used to characterize the over-dispersion found in count

data or when the mean for each Poisson observation is gamma distributed. In fact, for some of

these runs, the data are highly over-dispersed, i.e., 1=φ . Similar to the results shown for the

Poisson distribution, the bootstrap method provides a good estimate of the theoretical variance in

most cases, except for small sample sizes, where it slightly underestimate the true value. As

expected, with the increase in sample size, the bootstrap estimations perform better.


14

Table 2 Results for the Negative binomial simulation

Sample

Size Mean λ

IDPa φ

Theoretical Sum

)(χS

Theoretical Varianceb

Resampling Variance

))(( * χSVar

Ratio r c

mean variance MSEd

20

1 2 20 30 29.90 1.00 0.32 0.32

5 1 100 600 575.49 0.96 0.29 0.29

5 2 100 350 349.59 0.99 0.22 0.22

20 2 400 4400 4165.20 0.95 0.20 0.20

50

1 2 50 75 76.48 1.02 0.12 0.12

5 1 250 1500 1489.40 0.99 0.13 0.13

5 2 250 875 838.45 0.96 0.08 0.08

20 2 1000 11000 11192 1.02 0.10 0.10

100

1 2 100 150 153.51 1.02 0.05 0.05

5 1 500 3000 2942.3 0.99 0.07 0.07

5 2 500 1750 1748.3 1.00 0.06 0.06

20 2 2000 22000 21616 0.98 0.04 0.04

Note: a IDP: Inverse dispersion parameter of a Poisson-gamma model (φ ) b ( ) 2Var Y λ λ φ= + c The ratio of bootstrap variance to theoretical variance ( )(/))(( * χχ SSVar ). 100 r values are generated in each parameter setting (or each row) from the simulation. The mean and variance are calculated based on these 100 values. d MSE: Mean Square Error. MSE = (Bias)2 + Variance.

The simulation results for the third series (a mixture of Poisson and NB distributions) are

summarized in Table 3. This table shows that for every scenario evaluated, the bootstrap method

properly estimated the variance as the predicted values are very close to the theoretical values. At

this point, it is probably safe to state that the bootstrap method is a reliable approach for

estimating the sample mean and sample variance of observed (and predicted) data.


15

Table 3 Simulation results for the mixed-distribution simulation

Proportion Theoretical Sum

Theoretical Variance

Mean Sample Suma

Resampling Variance

))(( * χSVar

Ratio r c

mean variance

Scenario 1b N1=50

N2=50 2000 12000 1987 11653 0.98 0.06Scenario 2

N1=25 N2=75 2000 7000 1998 6924 0.99 0.12

Scenario 3 N1=75 N2=25 2000 17000 1990 16707 0.98 0.05

Note: a The sample sum reflects the use of equation ˆ ˆ( )jλ λ=∑ or ˆ ˆ( )jπ π=∑ b In each scenario, the sample size is 100, which consists N1 data from the Negative Binomial distribution with mean 20λ = and inversed dispersion parameter 2.0φ = , and N2 data from the Poisson distribution with mean 20λ = . 100 simulations are conducted for each scenario. c The ratio of bootstrap variance to theoretical variance ( )(/))(( * χχ SSVar ). 100 r values are generated in each parameter setting (or each row) from the simulation. The mean and variance are calculated based on these 100 values.

In summary, the simulation results have shown that the non-parametric bootstrap resampling

method is a simple and efficient approach for estimating the variance of random variables. This

method offers reliable estimates even when the sample size is relatively small. In the next

section, the bootstrap method is applied to observed data to examine whether the variance

traditionally computed in before-after studies can be replicated.

5. Application of the Proposed Methods to Observed Data

This section describes the application of the proposed methods to observed before and after

data. The methods were applied to two datasets. The first dataset contained crash data collected

in North Carolina for evaluating the safety effects of red-light cameras at signalized

intersections. The second dataset included crash data collected for a study that examined the


16

safety effects of converting signalized and unsignalized intersections to modern roundabouts. It

is important to point out that this part of the analysis only focuses on the estimation of the

variances and does not address nor discuss the conclusions reported in the original manuscript

from which the data were extracted.

6.1 1ST Dataset – Red Light Running Crash Data

Red-light cameras have become an increasingly popular strategy for improving traffic safety

at signalized intersections. This strategy aims at reducing the number of incidences associated

with red-light running, with the hope that a reduction in the number of crashes occurring at

signalized intersections would be observed. This topic has been studied extensively by a large

number of researchers (Hooke et al., 1996; Retting, 1999; CDOT, 2000; Radalj, 2001; Lum and

Wong, 2002; NCHRP, 2003; Flannery and Maccubbin, 2003). In most cases, they have reported

a positive effect where a reduction in the number of motor vehicle crashes caused by red-light

running has been noted. However, it has recently been reported that the installation of cameras

have also contributed to an increase in rear-end collisions, which according to some, reduces the

overall effectiveness of this strategy (Council et al., 2005; Washington and Shin, 2006).

For this analysis, data collected by the City of Charlotte Department of Transportation for

estimating the safety effects of red-light cameras in Charlotte, NC were analyzed (CDOT, 2000).

The data are summarized in Table 4. The data include information on the location where the

cameras were installed as well as the crash frequencies (fatal injuries, injury types A, B and C,

and property damage only) for the before and after periods. The CDOT installed cameras at 17

intersections and collected crash data three years before and after the installation. Table 4 shows

that a reduction in motor vehicle crashes (right-angle and rear-end) was not observed at every


17

intersection where they were installed, but the overall number of crashes decreased in the after

period.

Table 4 Crash data before and after the installation of red-light cameras in Charlotte, N.C.

(CDOT, 2000)

Location Crashes (3 years)

Location

Crashes (3 years)

Beforea Aftera Beforea Aftera Beatties Ford Rd./Hoskins Rd. 4 2 Westinghouse

Blvd./S.Tryon 23 11

Morehead St./College St. 29 10 Popla St./4th St. 24 20

Tyvola Rd./Wedgewood Dr.

27 12 Albemarle Rd. @ Harris Bv. 61 34

Morehead St./McDowell St. 18 10 Sharon Amity Rd. @

Central Av. 32 43

Brookshire Freeway/Hovis Rd. 44 28 Eastway Dr. @

Kilborne Dr. 25 27

11th St./Brevard St. 26 16 Fairview Rd. @ Sharon Rd 27 28

Arrowood Rd./Nations Ford Rd.

9 14 Idlewild Rd. @ Independence Bv. 33 25

N.Tryon St./Harris Blvd. 43 46 Randolph Rd. @

Sharon Amity Rd. 18 12

South Blvd./Archdale Dr. 25 29 Sum 468 367

Note: a Crash frequencies only include rear-end and right-angle crashes located on approaches

where the camera is located.

For this dataset, a naïve before-after study was conducted.2 The variance-to-mean ratios were

8.38 and 7.02 for the before and after periods, respectively. The crash data for both periods

showed a significant amount of over-dispersion. The inverse dispersion parameter was estimated

via the maximum likelihood estimation (MLE) techniques for both periods; note that the

2 The change in traffic flow was not applied here, since this information was not available. This does not affect the conclusion of the analysis or the premise of the paper.


18

variance of the parameterization used in this research is the following ( ) φμμ 2+=YVar , as

described in Equation (5). The inverse dispersion parameters were estimated to be 4.75 and 3.26

for the before and after periods, respectively.

For calculating the variances ( }ˆ{λVar and }ˆ{πVar ) (note: }ˆ{πVar here is estimated using

observed crash counts), the traditional, the parametric and the non-parametric bootstrap methods

were applied to the data. The results are summarized in Table 5. From this table, it can first be

observed that the mean values (λ̂ and π̂ ) estimated by all methods are, as expected, exactly the

same. On the other hand, the variances ( }ˆ{λVar and }ˆ{πVar ) estimated by the traditional

method differs significantly from those estimated by the parametric and non-parametric methods.

For instance, the variance *ˆ{ } 2800Var λ = calculated by the parametric method is much higher

than the value obtained by the traditional method ( 367)(}ˆ{ == χλ SVar ). This method

underestimated both variances ( }ˆ{λVar and }ˆ{πVar ). The inferences associated with index of

effectiveness θ̂ have shown similar results, in which the variances of θ̂ calculated by the

proposed method are higher than the traditional method.


19

Table 5 Comparisons of results for the red-light camera data

Statistics TraditionalMethod

Parametric Method

Bootstrap Method

π̂ 468 468 468

λ̂ 367 367 367

δ̂ 101 101 101

}ˆ{πVariance 468 3174 2939

}ˆ{λVariance 367 2800 2400

}ˆ{δVariance 835 5974 5339

θ̂ 0.783 0.773 0.774

}ˆ{θVariance 0.003 0.020 0.018

The results in Table 5 also show that ˆ{ }Var δ and ˆ{ }Var θ are also affected by the selection of

the method. In both cases, the variances are much larger for the proposed methods than with the

current method.

6.2 2nd DATASET - ROUNDABOUT CONVERSION

Many studies have shown that converting conventional intersections to modern roundabouts

could significantly reduce motor vehicle crashes and injuries (Schoon and Minnen, 1994;

Troutbeck, 1993; Elvik et al., 1997; Flannery and Elefteriadou, 1999; Persaud et al., 2001).

Persaud et al. (2001), from which the data used in this study were extracted, analyzed the safety

effects of converting stop-controlled and signalized intersections to modern roundabouts. They

collected data in seven U.S. states and performed a before-after study using the EB technique. In

this study, the data collected for single lane, urban stop controlled intersections were used and a

naïve before-after study was performed on the data. The characteristics of the data are


20

summarized in Table 6. A total of eight intersections were converted to a modern roundabout.

The data include the total number of crashes per year.

Table 6 Crash data before and after the conversion of modern roundabouts (Persaud et al., 2001)

Jurisdiction

Annual Crash Counts

Predicted After

(crashes/year)

Observed After

(crashes/year) Bradenton Beach, FL 1.7 0.2 Fort Walton Beach, FL 8 2.0 Gorham, ME 6 3.2 Hilton Head, SC 16 2.3 Manchester, VT 0.4 0.4 Manhattan, KS 3 0.0 Montpelier, VT 1.2 0.3 West Boca Raton, FL 1.5 1.7 Sum 37.8 10.1

Note: the values in this table are already adjusted for the differences in time periods and traffic flows.

Results from the before-after study and the proposed method are shown in Table 7. This table

shows that the variances calculated by both proposed methods are still larger than the sum of

crashes over the entire group of entities for the before period. The estimated variances for all the

methods are very similar for the after period. However, the bootstrap output shows that the

variance is actually slightly lower than the mean (e.g., under-dispersion) for λ̂ , which is

explained by the very small sample size and low sample mean values. This is not uncommon for

this type of dataset, as documented by Oh et al. (2006) and Lord et al. (2009). The inverse

dispersion parameter was estimated using the method of moments (MM); the MLE did not


21

converge for this dataset (see Lord, 2006 about this issue). The inverse dispersion parameters

were equal to 0.98 and 9.04 for the before and after periods, respectively. For the latter, it is most

likely over-estimated, again caused by the low sample mean problem (Lord, 2006). Finally,

Table 7 shows that the values θ̂ for the proposed methods are very comparable, but somewhat

lower than the traditional method (0.260). Thus, it is possible that the traditional method could

have underestimated the effectiveness of roundabout conversions (only for the naïve before-after

study).

Table 7 Comparisons of results for roundabout conversions

Statistics Traditional Method

Parametric Method

Bootstrap Method

π̂ 37.8 37.8 37.8

λ̂ 10.1 10.1 10.1

δ̂ 27.7 27.7 27.7

}ˆ{πVariance 37.8 220.6 189.7

}ˆ{λVariance 10.1 11.5 9.3

}ˆ{δVariance 47.9 232.1 199.0

θ̂ 0.260 0.231 0.236

}ˆ{θVariance 0.008 0.011 0.010

6. Discussion

The results of this study indicate that important limitations exist with the current approach for

conducting traditional before-after studies. First, the results show that a large discrepancy exists

between }ˆ{λVar and ˆ{ }Var π estimated with the traditional method and the values calculated by

the parametric and non-parametric bootstrap resampling methods. The traditional method uses

observed data to approximate the expected value of a Poisson distribution and treat these values


22

as known quantities for estimating the variance. However, the true variance is actually unknown,

as noted by Hauer (1997). Thus, using observed counts as a substitute for the variance is not

valid and will underestimate the true variance for each entity.

Second, as shown in the simulation results and the empirical analyses, the variances

estimated using the traditional method is likely to be underestimated. This underestimation could

have an important effect on the safety effectiveness of interventions or treatments. For instance,

some interventions that would have been identified as being statistically significant (e.g., the

95%-percentile confidence interval does not include 1) using the traditional method could in

reality be non-significant. This problem is even more important in the light of the methodology

proposed by Hauer (2005b), which will be used to include or exclude accident modification

factors (AMFs) in the upcoming Highway Safety Manual (HSM). This methodology uses

information about the confidence intervals as well as the stability of the estimated factor of

effectiveness (θ ). Thus, a biased estimate of the variance will affect whether an AMF should be

included or not in the HSM.

Third, the assumption that crash data are Poisson distributed, even for a single entity,

needs to be re-examined in the context of the conditional properties of a random variable

described in this manuscript. As discussed by Lord et al. (2005), crash data are in fact the results

of Bernoulli trials with unequal probability of events. The Poisson distribution and mixed

distributions, such as the Poisson-gamma or Poisson-lognormal, are in effect used as

approximate distribution for analyzing motor vehicle crashes. It is suggested to expand the work

of Nicholson and Wong (1993) and Lord et al. (2005) for the analysis of crash counts at single

entities.


23

7. Summary and Conclusions

This paper has documented the application of parametric and non-parametric bootstrap

resampling methods for estimating the variance in before-after studies. This paper was motivated

on the grounds that to estimate the variance, crash data on each entity that is part of the analysis

are used as a substitute for estimating the variance. Although the actual variance is unknown, as

noted by Hauer (1997), this assumption is used to simplify the computation of the variance. The

literature on this topic has shown that the variance should be estimated using the conditional

properties of random variables.

The objectives of this paper were to evaluate whether or not the assumption that crash data

should be used as a direct substitute to estimate the variance is valid, and if not, to examine how

this may affect the estimation of the variance for calculating the inferences of parameters used in

traditional before-after studies. To accomplish the objectives of this study, a parametric method

that incorporated the conditional properties of a random variable, and a non-parametric bootstrap

method were investigated to evaluate this assumption. The bootstrap method was initially

applied to data simulated using a Poisson distribution, a Negative Binomial distribution, and a

mixture of these two distributions. Then, the traditional and the proposed methods were applied

to two datasets containing observed before and after crash data taken from the literature.

The results of this study show that observed data should not be used as a substitute for the

variance, even if the entities are assumed to follow a Poisson distribution. Consequently, the

estimated variances are likely to be underestimated when the traditional method is used. The

proposed methods can overcome this problem and provide a more accurate estimation of the

variance. It is hoped that the results of this study will foster researchers to continue further on


24

work conducting before-after studies, the characteristics of the Poisson distribution, and

conditional probabilities applied in highway safety research.

Acknowledgements

The authors greatly acknowledge the comments and suggestions provided by Dr. Michael

Longnecker from the Department of Statistics at Texas A&M University on the conditional

properties of random variables, as described in this study. The authors also thank Dr. Alan

Nicholson from the University of Canterbury for providing input relevant for this work.

References

Anscombe, F.J. (1949). The Statistical Analysis of Insect Counts Based on the Negative

Binomial Distributions. Biometrics, 5, 165-173.

Agresti, A. (2002). Categorical Data Analysis. John Wiley & Sons, Hoboken, N.J.

Bolstad, B.M. (2004). Introduction to Bayesian Statistics. Wiley-Interscience, Hoboken, N.J.

Casella, G., & Berger, R.L. (1990). Statistical Inference, Duxbury Press, CA.

Charlotte Department of Transportation, (2000). SafeLight Charlotte: annual report: August 1999

– July 2000.

Clark, S.J., & Perry, J.N. (1989). Estimation of the Negative Binomial Parameter k by Maximum

Quasi-Likelihood. Biometrics, 45, 309-316.

Cook, R.J., & Wei, W. (2001). Selection effects in randomized trials with count data. Statistics in

Medicine, 21, 515-531.

Council, F., Persaud, B., Eccles, K., Lyon, C., & Griffith, M. (2005). Safety evaluation of red-

light cameras: executive summary. Report no. FHWA HRT-05-049. Washington, DC:

Federal Highway Administration.


25

Diggle, P.J., Liang, K.Y., & Zeger, S.L. (2002). Analysis of Longitudinal Data. Oxford Univ.

Press Inc., New York, NY.

Efron, B., & Tibshirani, R. (1993). An introduction to the bootstrap. Chapman and Hall, New

York.

Elvik, R., Mysen, A.B., & Vaa, T. (1997). Traffic Safety Handbook (Norwegian). Institute of

Transport Economics (TØI), Oalo, Norway.

Fisher, R.A. (1941). The Negative Binomial Distribution. Annals of Eugenics, 11, 182-187.

Flannery, A., & Elefteriadou, L., (1999). A review of roundabout safety performance in the

United States. Proceedings of the 69th Annual Meeting of the Institute of Transportation

Engineers (CD-ROM), Institute of Transportation Engineers, Washington, D.C..

Flannery A, & Maccubbin R. (2003). Using meta analysis techniques to assess the dafety effect

of red light running cameras. TRB 2003 Annual meeting paper, Washington DC:

Transportation Research Board.

Hauer, E. (1997). Observational Before-After Studies in Road Safety: Estimating the Effect of

Highway and Traffic Engineering Measures on Road Safety. Oxford, England: Pergamon

Press, Elsevier Science Ltd.

Hauer E. (2005a). Cause and effect in observational cross-section studies on road safety. Draft

report. HSIS, Washington DC: Federal Highway Administration.

Hauer, E. (2005b). A decision Based Approach to Include or Exclude AMFs. Draft Research

Paper. Toronto, Ont.

Hall, P. (1989). Bootstrap methods for constructing confidence regions for hands.

Communications in Statistics. Stochastic Models, 5(4), 555-562.


26

Haynor, D.R., & Woods, S.D. (1989). Resampling estimates of precision in emission

tomography. IEEE Transactions on Medical Imaging, 8, 337-343.

Hooke A, Knox J, & Portas, D. (1996). Cost benefit analysis of traffic light & speed cameras.

Police Research Series Paper, 20, 57.

Kawano, H., & Higuchi, T. (1995). The bootstrap method in space physics: error estimation for

the minimum variance analysis. Geophysical Research Letters, 22(3), 307-310.

Lehmann, E.L. (1983). Theory of point estimation. Wiley, New York, NY.

Laird, N.M., & Ware, J.H. (1982). Random-effects models for longitudinal data. Biometrics,

38(4), 963-974.

Lord, D. (2006). Modeling motor vehicle crashes using Poisson-gamma models: Examining the

effects of low sample mean values and small sample size on the estimation of the fixed

dispersion parameter. Accident Analysis & Prevention, 38(4), 751-766.

Lord, D., S.R. Geedipally, and S. Guikema (2009) Extension of the Application of Conway-

Maxwell-Poisson Models: Analyzing Traffic Crash Data Exhibiting Under-Dispersion.

Working Paper, Zachry Department of Civil Engineering, Texas A&M University, College

Station, Texas.

Lord, D., Washington, S.P., & Ivan, J.N. (2005). Poisson, Poisson-Gamma and zero inflated

regression models of motor vehicle crashes: Balancing statistical fit and theory. Accident

Analysis & Prevention, 37(1), 35-46.

Lum K, & Wong, Y. (2002). Effects of red light camera installation on driver behaviour at a

signalised cross-junction in Singapore. Road &Transport Research, 11 (3).

National Cooperative Highway Research Program. (2003). Impact of red light camera

enforcement on crash experience. National Cooperative Highway Research Program 57.


27

Nicholson A, & Wong Y. (1993). Are accidents Poisson distributed? A statistical test. Accident

Analysis and Prevention, 25 (1), 91-97.

Noland R.B. (2003). Traffic fatalities and injuries: the effect of changes in infrastructure and

other trends. Accident Analysis and Prevention, 35, 599-611.

Oh, J., S.P. Washington, and D. Nam (2006) Accident Prediction Model for Railway-Highway

Interfaces. Accident Analysis and Prevention, 38 (2), pp. 346-56.

Park E.S., Park J., & Lomax T. (2009). Fully Bayesian multivariate approach to before-and-after

safety evaluation. TRB 2009 Annual meeting paper, Washington DC: Transportation

Research Board.

Persaud B.N., Retting, R., Garder, P., & Lord, D. (2001). Observational before-after study of

U.S. roundabout conversions using the empirical Bayes method, Transportation Research

Record, 1751, 1-8.

Persaud B., Lan B., Lyon, C., & Bhim, R. (2009).Comparison of empirical Bayes and full Bayes

approaches for before-and-after road safety evaluations. TRB 2009 Annual meeting paper,

Washington DC: Transportation Research Board.

Persaud, B.N., McGee, H., Lyon, C., & Lord, D. (2003). Development of a Procedure for

Estimating the Safety Effects for a Contemplated Traffic Signal Installation. Transportation

Research Record ,1840, 96-103.

R Development Core Team. (2007). R: A language and environment for statistical computing. R

Foundation for Statistical Computing. Vienna, Austria. ISBN 3-900051-07-0. Retrieved Jun

18, 2007, from http://www.R-project.org

Radalj, T. (2001). Evaluation of effectiveness of red light camera programme in Perth. Road

Safety Research, Policing and Education Conference, Melbourne, Australia.


28

Retting R, Williams R, Farmer C, & Feldman A. (1999). Evaluation of red light camera

enforcement in Oxnard, California. Accident Analysis and Prevention, 31, 69-74.

Rich, H. (2001). Using the Bootstrap with small data sets: the smoothed Bootstrap. RSS Matters,

Benchmarks Online, University of North Texas. Retrieved May 10, 2007, from

http://www.unt.edu/benchmarks/archives/2001/october01/rss.htm

Saradhi, V.V., & Murty, M.N. (2001). Bootstrapping for efficient handwritten digit recognition.

Pattern Recognition, 34(5), 1047-1056.

Schaik, G.V., Shoukri, M., Martin, S., Schukken, Y.H., Nielen, M., Hage, J.J., & Dijkhuizen,

A.A. (1999). Modeling the Effect of an Outbreak of Bovine Herpesvirus Type 1 on Herd-

Level Milk Production of Dutch Dairy Farms. Journal of Dairy Science, 82, 994-952.

Schoon, C., & van, M.J. (1994). The safety of roundabouts in the Netherlands. Traffic

Engineering and Control, 142–148.

Tarko, A. P., Eranky, S., & Sinha K.S. (1998). Methodological considerations in the

development and use of crash reduction factors. Preprint CD. 77th Annual Meeting of the

Transportation Research Board.

Tauxe, L., Kylstra, N., & Constable, C. (1991). Bootstrap statistics for Palaeomagnetic data.

Journal of Geophysical Research, 96, 11723-11740.

Troutbeck, R. J. (1993). Capacity and design of traffic circles in Australia. Transportation

Research Record 1398, TRB, National Research Council, Washington, D.C., 68-74.

Washington, S.P., & Shin, K. (2006). Impact of red light cameras on safety in Arizona.

Proceedings of the 85th Annual Meeting of the Institute of Transportation Engineers (CD-

ROM), Institute of Transportation Engineers, Washington, D.C.


29

Willson, L.J., Folks, J.L., & Young, J.H. (1984). Multistage Compared with Fixed-Sample-Size

Estimation of the Negative Binomial Parameter k. Biometrics, 40, 109-117.

Zhang, Y., Ye, Z., and Lord, D. (2007). Estimating the Dispersion Parameter of the Negative

Binomial Distribution for Analyzing Crash Data Using a Bootstrapped Maximum

Likelihood Method. Transportation Research Record 2019, pp. 15-21.

Zoubir, A.M., & Iskander, D.R. (2004). Bootstrap techniques for signal processing. Cambridge

University Press.

Documents

Running head: ESTIMATING THE VARIANCE · Estimating the Variance 2 Abstract Problem: To simplify the computation of the variance in before-after studies, it is generally assumed that