Model misspecification in Data Envelopment Analysis

© J.C. Baltzer AG, Science Publishers

Chapter 12

Model misspecification in Data Envelopment Analysis

Peter SmithDepartment of Economics and Related Studies,

University of York, York YO1 5DD, UK

E-mail: [email protected]

“The fact is, there are only two qualities in the world: efficiency and inefficiency,and only two sorts of people: the efficient and the inefficient.”

George Bernard Shaw, John Bull’s Other Island, 1907.

The use of Data Envelopment Analysis for estimating comparative efficiency has becomewidespread, and there has been considerable academic attention paid to the development ofvariants of the basic DEA model. However, one of the principal weaknesses of DEA is that– unlike statistically based methods – it yields no diagnostics to help the user determinewhether or not the chosen model is appropriate. In particular, the choice of inputs and out-puts depends solely on the judgement of the user. The purpose of this paper is to examinethe implications for efficiency scores of using a misspecified model. A simple productionprocess is set up. Simulation models are then used to explore the effects of applyingmisspecified DEA models to this process. The phenomena investigated are: the omission ofsignificant variables; the inclusion of irrelevant variables; and the adoption of an inappro-priate variable returns to scale assumption. The robustness of the results is investigated inrelation to sample size; variations in the number of inputs; correlation between inputs;and variations in the importance of inputs. The paper concludes that the dangers of mis-specification are most serious when simple models are used and sample sizes are small. Insuch circumstances, it is concluded that it will usually be to the modeller’s advantage to erron the side of including possibly irrelevant variables rather than run the risk of excludinga potentially important variable from the model.

Keywords: Data Envelopment Analysis, model specification, simulation.

1. Introduction

The development of the theory of Data Envelopment Analysis (DEA) proceedsapace. However, as Seiford and Thrall [19] note, surprisingly little attention has beenpaid to the practical problem of how to develop a satisfactory DEA model. There

Annals of Operations Research 73(1997)233–252 233

exists no accepted modelling strategy for the DEA user, and there has been littlesystematic effort to examine the impact of misspecification on model results. Instead,the emphasis has been on exploring the sensitivity of results to data errors [9]. To thatend, the development of stochastic data envelopment analysis offers a potentially richanalytic framework [20].

The relatively undeveloped state of methodology for the DEA user can becontrasted with mainstream econometrics, in which considerable thought has beengiven to methodology [18]. In particular, econometricians have developed a substan-tial battery of devices to test whether a putative model is correctly specified. Ofcourse, the need to specify a functional form means that tests for specification mustplay a central role in econometric studies [15], and tests for misspecification havebecome standard in most econometric software packages [16]. However, even thoughno functional form is specified in the DEA model, it is clear that model specificationmust be a central concern. In particular, the user must specify a set of inputs andoutputs, and must make a choice regarding returns to scale. Wrong choices in relationto these features are likely to compromise the value of the analysis.

Moreover, the problem of misspecification in DEA is particularly intractablebecause – unlike econometric methods of frontier estimation – DEA offers the userno diagnostics with which to judge the suitability of the chosen model. Because ofthe lack of model selection procedures, it therefore seems important to offer the DEAuser some guidance on the likely impact of the various sorts of specification errorthat are likely to occur. An understanding of the implications of misspecification mayhelp the user avoid the most serious consequences of such misspecification, and is afirst necessary step in the development of a workable modelling strategy for the DEAuser. Without such a strategy, the user is likely to be left to flounder in a mess ofmany alternative specifications telling different stories [14].

Some exploratory work has been undertaken on the robustness of DEA resultsto alternative specifications. Ahn and Seiford [1] systematically document alternativemodel specifications, and explore their implications for efficiency estimates in thehigher education sector. They conclude that DEA results are robust with respect tomodel specification. Most DEA researchers who have been concerned with examin-ing the robustness of results to specification have tended to adopt a less rigorousapproach, perhaps testing a variety of models in a rather ad hoc fashion. Valdmanis[22] presents one such study, in which ten different specifications of a hospital effi-ciency model were tested. On average, the results in that study were also found to befairly robust to alternative specifications.

However, when results are more dependent on specification, the researcher hasavailable little guidance on how to choose between competing models, and noevidence on the consequences of misspecification. Moreover, although the studiesnoted above were not concerned with the extent to which the ranking of individualhospitals varied between models, in most practical applications of DEA, managers ofindividual decision making units (DMUs) are likely to be profoundly interested in

P. Smithy Model misspecification in DEA234

the outcome for their own DMU. Thus, even if average efficiencies do not vary sub-stantially between models, it may also be important to examine the extent to whichdifferent specifications favour different DMUs.

An alternative means of testing robustness – for a given specification – is tocompute “cross-efficiencies” [13]. These indicate the efficiency of a DMU usingthe preferred weightings for other DMUs, rather than its own preferred weighting.Analysis of cross efficiencies therefore yields useful information on the sensitivity ofa DMU’s results to a variety of feasible weighting systems. However, it says littleabout the merits of alternative specifications.

The purpose of this paper is to take the first step towards the development of amodelling strategy by examining the impact of a variety of misspecifications on theresults emerging from a simple DEA model. Underlying the analysis are some simplebut plausible productivity processes, the form of which is known. The processes aredescribed in the next section. A Monte Carlo simulation is used to generate a largenumber of observations. Section 3 reports the results of applying the correctly speci-fied DEA model to samples of various sizes. As expected, DEA estimates of efficiencyare closest to true levels of efficiency when sample sizes are large and the productionprocess is simple. The results cast some quantitative light on the relative importanceof these two determinants of the quality of the DEA model. The correctly specifiedDEA models are then used in section 4 as benchmarks for investigating various formsof misspecification. The phenomena investigated are omission of a salient variable,inclusion of an irrelevant variable, and incorrect returns to scale assumptions. Section 5explores the robustness of the results with respect to some of the assumptions under-lying the production model, and section 6 draws some practical conclusions for theDEA model builder.

2. The production model

A variety of forms of the DEA model exists [12]. In order to facilitate exposi-tion, this paper adopts the following representation, using the notation of Ali andSeiford [2]. The efficiency of decision making unit 0 (DMU 0) is given by θ0 = 1yφ0,where φ0 is the solution to the linear programme1)

maximize

subject to

and

0φ

λ φ

λ

λ

j rj r rj

n

j i j i ij

n

j i r

y s y r s

x e x i m

e s i j r

− = = …

+ = = …

≥ ∀

=

=

∑

∑

0 01

01

1

1

0

, , ,

, , ,

, , , ,

(1)

1) As explained in [2], the problem set out in (1) is the first stage in a two-stage model. It is necessaryto solve a second linear programme to ensure that fully efficient DMUs are identified. For presenta-tional purposes, this refinement is omitted here, but was fully implemented in all computational work.

P. Smithy Model misspecification in DEA 235

where xj is the m-vector of inputs and yj the s-vector of outputs for DMU j. Inintuitive terms, the programme searches for the comparison group – formed byweighting DMU j by a coefficient λj – which uses no more inputs than DMU 0 andwhich maximizes outputs in relation to those of DMU 0. This is the output-orientedversion of the original DEA model proposed by Charnes et al. [10,11], which implic-itly presumes the existence of constant returns to scale.

In this paper we utilize a known production process, in which a number of inputs,represented by vector x, are used to produce a single output y. That is, s = 1. Theoutput of unit j is given by

where Y, b, X and E are natural logarithms of y, β, x and η, and E ∈(–∞, 0).In order to proceed, it is necessary to specify a statistical distribution for the

variables Y, X and E. A number of approaches have been used in previous DEA MonteCarlo simulations [6]. We choose to assign normal distributions with unit variances toeach Xi . In addition, a covariance structure must be constructed. In most practicalapplications of DEA, inputs are highly correlated because they are all related to thescale of operations of the DMU. As a result, for most of the results presented herewe assume a covariance of 0.8 between all inputs. However, in order to explorethe importance of this assumption, we report in section 5 the results of assuming nocorrelation between inputs. Throughout, efficiency E is drawn from a half-normal dis-tribution with unit variance, and is assumed to be uncorrelated with any X variable. Itmust be emphasized that this assumption is made solely to ensure that a realistic rangeof inefficiency values is generated, and that it is not intrinsic to the results that follow.

A major impediment to judging the practical importance of theoretical results inDEA is that the data sets used to illustrate the findings are usually small, and it isoften impossible to disentangle the effects caused by the phenomenon of interest fromthose due to the idiosyncrasies of the data. In order to overcome this problem, thisstudy uses Monte Carlo simulation to yield a series of 10,000 observations of X andE for each model under investigation. The associated values of Y are computed using(3). For the purposes of estimating efficiencies, these series are then divided up intosmall subsets, and the average efficiencies yielded by DEA are reported.

For most of the paper we assume equal importance of inputs, so with m inputs,each parameter αi is set equal to 1ym, and β is set equal to one throughout. Again,

y xj ij j ii

m

i

mi= =

==∑∏β η αα ,1

11

(2)

where ηi ∈[0,1] is the efficiency of DMU j and β and α are known parameters.Equation (2) represents the familiar Cobb–Douglas functional form with constantreturns to scale and the incorporation of a technical inefficiency term η. An alterna-tive additive representation is

Y b X Ej i ij ji

m

= + +=∑ α , ( )

1

3


in section 5 we test the implications of alternative assumptions. All variables areexponentiated to yield the values of y and x – which are to be used in DEA – and η,the true efficiencies with which the DEA results are to be compared. The process isrepeated for five different models, using values of m (the number of inputs) rangingfrom 2 to 6.

There is of course no suggestion that real-life production processes conformto this rather neat form. In addition, there are in practice likely to be stochasticperturbations to data caused by unknown environmental variables or measurementerror. And it may be that observed distributions of inputs do not in practice conformto the assumptions of normality. However, non-parametric methods such as DEA arevaluable precisely because in practice processes appear to exhibit less regularity thanthat assumed when using standard econometric methods. The purpose of this paper isto shed light on the distortions in efficiency estimation that can arise when using DEA,and to that end the use of a simple and well-understood production process is appro-priate. The reader is invited to extrapolate the results of this stylized scenario to morerealistic settings.

3. DEA results using correctly specified models

In this section, we report the estimates of efficiency yielded by DEA when themodel is correctly specified. Clearly it is meaningless to construct a DEA model withthe entire population of 10,000 observations, as such samples never exist in practice.Instead, each observation is assigned to a single subset of DMUs, and a DEA modelconstructed for each subset in turn.2) Four realistic sizes of subset are investigated:10, 20, 40 and 80. Thus, for example, 1000 DEA models are estimated for the entirepopulation divided into subsets of size 10. These small subsets are then paired to yieldsubsets of size 20, and efficiencies recomputed using the consequent 500 models. Thisprocess is repeated for subset sizes 40 and 80. The subsets are therefore nested sothat, if two DMUs are in the same subset of, say, size 10, they are always assigned tothe same subsets of larger size. This methodology allows the effects of realistic samplesizes to be examined, but at the same time the presentation of averages derived froma large number of models eliminates most of the random variability intrinsic to studiesusing limited datasets. The nested nature of the samples used means that, as samplesize increases, a DMU’s efficiency can never increase, because its comparison set isalways augmented by new DMUs [17,21]. In effect, a larger sample size increasesthe possibility of encountering DMUs close to the production frontier, and thereforethe DEA frontier approaches the true frontier asymptotically as sample size increases[3].

Results for the five correctly specified models are summarized in table 1. In thisand all subsequent tables geometric means of the efficiencies of all 10,000 DMUs are

2) In this paper, a DEA model is construed as the entire set of n linear programmes required to estimatethe efficiencies of the n DMUs in the sample.


Table 1

Actual and DEA average efficiencies with well specified models.

True DEA efficiencies with sample:

efficiency 10 20 40 80

Two Efficiency 0.4516 0.5906 0.5400 0.5076 0.4871inputs Overestimate (%) NA 30.8 19.6 12.4 7.9

Three Efficiency 0.4505 0.6464 0.5943 0.5499 0.5215inputs Overestimate (%) NA 43.5 31.9 22.1 15.8

Four Efficiency 0.4501 0.6811 0.6252 0.5817 0.5475inputs Overestimate (%) NA 51.3 38.9 29.2 21.6

Five Efficiency 0.4545 0.7149 0.6582 0.6147 0.5777inputs Overestimate (%) NA 57.3 44.8 35.2 27.1

Six Efficiency 0.4522 0.7296 0.6742 0.6294 0.5930inputs Overestimate (%) NA 61.3 49.1 39.2 31.1

reported. The results are in line with expectations. In the deterministic setting assumedhere, using a convex production function, a well-specified DEA model will alwaysoverestimate efficiency. However, the extent of the overestimate is highly dependenton sample size. For example, using the two input model, the average overestimatereduces from an average of 31% with samples of size 10 to just 8% as sample sizeincreases to 80. Note also that, as the complexity of the production process increases(as indicated by the number of inputs), the success with which DEA can indicatetrue efficiencies diminishes. Thus, even with large sample sizes, DEA overestimatestrue efficiencies by 31% in the six input model. This result graphically illustratesthe difficulty that DEA encounters with complex production processes. The accuratemodelling of a surface in seven dimensional space requires far more data points thanthe equivalent in three dimensions, and results in a considerable increase in extremalobservations. Detailed inspection of the relationships between true efficiency andDEA efficiency shows that the principal reason for the overestimate is the largenumber of DMUs designated 100% efficient by DEA, even though their true efficiency(η) might be considerably less than unity.

4. DEA results using misspecified models

In this section, we report the changes to DEA efficiency estimates brought aboutby various forms of misspecification. To this end, the models described above are re-estimated with various defective assumptions, namely: one of the inputs is omittedfrom the model; an irrelevant input is incorporated into the model; and an assumption


of variable returns to scale is made. Throughout this section, the benchmarks againstwhich the misspecified models are judged are the DEA efficiencies from the correctlyspecified models set out in table 1.

4.1. Omission of an input

The first DEA misspecification to be examined is the omission of an input vari-able xm.3) This has the effect of removing one of the input constraints from model (1),and therefore may result in a reduction in estimated efficiency θ0. However, it willonly cause a change in the estimate of efficiency if the associated slack variable em

was zero in the original solution: that is, if the relevant constraint was binding. Otherthings being equal, the probability that a reduction in efficiency will arise will behighest for DMUs exhibiting low values of the omitted input in relation to the inputsremaining in the model, as these will set the toughest input requirements for theircomparison group. Indeed, pursuing this argument, the lower in relation to outputis the DMU’s consumption of the omitted variable, the greater is likely to be itsreduction in DEA efficiency rating arising from the variable’s omission. Conversely,other things being equal, the estimated efficiencies of DMUs with high values inthe omitted variable are likely to be least affected by the variable’s omission.

This phenomenon can be illustrated graphically. The top part of figure 1 indi-cates a DEA isoquant I* I * for a two-input model, with input measures standardizedfor a given level of output y* . Three efficient DMUs are shown: E1, E2 and E3. Thelower diagram shows a section Oabc through the efficient DEA frontier, standardizedfor a given level of the second input, say x2

*. It shows the implicit link between x1 andy, and indicates the expected diminishing marginal product of x1. The omission ofvariable x2 from the model has the effect of shifting the function to OP. Note,therefore, that it is DMUs employing relatively high levels of x1 (the retained input)in relation to x2 (the omitted input) that are likely to suffer the largest efficiencyreductions when judged in relation to the misspecified production function.

Simulation results confirm this theoretical analysis. For each of the six models,table 2 gives the proportion of DMUs experiencing a reduction in estimated efficiencyarising from the omission of xm. As expected, the probability that input constraint mis binding in the correct model decreases as the number of inputs increases, so thesimpler models display the highest proportion of DMUs affected by the mis-specification (95% with large samples). Also note that numbers affected increase withsample size. This is because – using the correctly specified model – the larger samplesizes result in fewer cases of non-binding input constraints.

Table 2 also indicates the average reduction in estimated efficiency broughtabout by the misspecification, and therefore gives an indication of the magnitude of

3) Because all α i are equal in the models used in this section, the imput chosen for omission is immate-rial. The implications of different assumptions about the values of α i are explored in section 5.


the aggregate distortion. It shows that in some circumstances the underestimates canbe substantial, but that they are highly dependent on sample size and the complexityof the model, with the largest distortions arising in large sample, simple models. Forexample, with the simplest two input model, the average efficiency underestimatescompared with the correctly specified DEA model range from 30% (sample size 10)to 41% (sample size 80). As the complexity of the model increases, the average impactof variable omission diminishes. With six inputs, the omission of an input results inunderestimates of DEA efficiency of between 4% (small samples) and 6% (for largesamples).

It is noteworthy that, for the two input case, the misspecified model is comprisedof one input and one output, and is effectively a partial performance indicator of thesort generated by the traditional ratio analysis of corporate performance [8]. Theresults in table 2 therefore give some indication of the distortions that arise from theuse of such imperfect measures of performance, and reinforce the desirability of usingmore comprehensive models of performance, such as those offered by DEA.

Figure 1. Impact on efficient frontier of omitting input x2.


Table 2

Average reductions in DEA efficiency arising from omission of input.

DEA efficiencies with sample:

10 20 40 80

Two % affected 75.4 88.5 89.6 95.2inputs Average efficiency 0.4126 0.3542 0.3147 0.2863

Underestimate (%) – 30.1 – 34.4 – 38.0 – 41.2

Three % affected 48.8 63.1 74.6 82.1inputs Average efficiency 0.5594 0.4992 0.4538 0.4207

Underestimate (%) – 13.5 – 16.0 – 17.5 – 19.3

Four % affected 36.5 43.8 56.0 65.8inputs Average efficiency 0.6279 0.5679 0.5207 0.4827

Underestimate (%) -7.8 -9.2 -10.5 -11.8

Five % affected 30.0 41.0 51.7 57.5inputs Average efficiency 0.6775 0.6184 0.5727 0.5333

Underestimate (%) – 5.2 – 6.0 – 6.8 – 7.7

Six % affected 26.0 35.6 45.0 55.2inputs Average efficiency 0.7030 0.6442 0.5968 0.5572

Underestimate (%) – 3.6 – 4.4 – 5.2 – 6.0

4.2. Inclusion of an extraneous variable

The introduction of an additional input variable into the DEA model (1) increasesthe number of constraints, and therefore cannot reduce measured efficiency, andmay increase it. In this section, we introduce an irrelevant variable into the correctlyspecified model, and examine its impact on estimated efficiency. The extraneous vari-able Xm+1 is drawn from a normal distribution with unit variance, and is assumed tohave a covariance of 0.8 with all input variables.

Clearly, if using the original solution to (1) the new constraint is non-binding,the optimal solution to the model is unchanged, and therefore the efficiency estimateis unaltered. Other things being equal, the new constraint is most likely to be non-binding for those DMUs for which the value of the extraneous input assumes arelatively high value in relation to output. Conversely, a low value of xm+1 in relationto y increases the probability that the original solution to (1) is infeasible, and that theefficiency estimate increases.

Figure 2 illustrates the impact of adding an extraneous variable when the correctmodel requires only one input. The top part of the diagram gives the correct DEAproduction frontier, passing through efficient DMU P* . Rays are then drawn from theorigin through all the DMUs, and their intersections with the line y = y* (where y* isan arbitrary output level) indicate the ratio of input x1 to output y used. These


Figure 2. Impact on efficient frontier of extraneous input x2.

observations are transferred to the horizontal axis of the lower diagram. The verticalaxis represents the extraneous variable x2 in relation to output y. The isoquant esti-mated by DEA when the extraneous variable is incorporated is indicated by the curveI2I2. Note that P* remains efficient. However, the revised efficiency rankings of theremaining DMUs depends on the values of x2 they exhibit. In this particular example,DMU P1 is assigned the lowest value of x2yy and is therefore deemed efficient. Theefficiency of most of the other DMUs is also increased by the introduction of x2.

The simulation results, shown in table 3, indicating that, although a large numberof DMUs are affected by the incorporation of an extraneous input into the model, theeffect on efficiencies is on average small. For example, using the simplest two inputmodel, with sample size 10, average DEA efficiency estimates increase from 0.59 to0.63, an overestimate of 6.4%. The overestimate is even smaller for larger samplesizes and for more complex models.

Throughout sections 4.1 and 4.2, we have concentrated on the average distortionsbrought about by misspecification. However, the user may also be interested in the


Table 3

Increases in average efficiency arising from inclusion of extraneous input.


10 20 40 80

Two % affected 35.8 42.7 47.7 46.7inputs Average efficiency 0.6282 0.5714 0.5323 0.5054

Overestimate (%) 6.4 5.8 4.9 3.8

Three % affected 32.5 42.5 47.1 47.7inputs Average efficiency 0.6762 0.6183 0.5720 0.5398

Overestimate (%) 4.6 4.0 4.0 3.5

Four % affected 22.3 28.1 33.1 45.0inputs Average efficiency 0.7036 0.6474 0.6014 0.5646

Overestimate (%) 3.3 3.6 3.4 3.1

Five % affected 18.8 23.3 28.1 37.1inputs Average efficiency 0.7323 0.6756 0.6307 0.5923

Overestimate (%) 2.4 2.6 2.6 2.5

Six % affected 15.8 19.6 26.9 26.7inputs Average efficiency 0.7442 0.6885 0.6435 0.6062

Overestimate (%) 2.0 2.1 2.2 2.2

distortions in the ranking of DMUs brought about by the misspecification. In order togive an idea of ranking distortions in the two input case, table 4 reports the Spearmanrank correlation coefficients of the true efficiency ranks of DMUs with the ranks usingDEA estimates of efficiency. The analysis is confined to DMUs that were deemedinefficient using a sample size of 10. The table confirms the generally good perform-ance of correctly specified DEA, and the much larger distortion brought about byomitting a relevant variable than that induced by including an extraneous variable.Similar results were found using models with more inputs.

Table 4

Spearman rank correlation coefficients between true efficiencyand DEA efficiency measures (two-input case).

Sample size

10 20 40 80

Correctly specified DEA model 0.881 0.942 0.972 0.991

DEA model with omitted variable 0.606 0.648 0.636 0.644

DEA model with extraneous variable 0.746 0.848 0.915 0.952


4.3. Returns to scale assumption

The production process used in this paper assumes constant returns to scale. Thisis in line with the returns to scale assumption implicit in the basic DEA model givenby (1). However, Banker et al. [4] have shown that it is a simple matter to incorporatea variable returns to scale assumption into DEA by adding the constraint

λ jj

n

==∑ 1 4

1

( )

to the original linear programme. Constraint (4) is often invoked in practical DEAstudies, as it ensures that only interpolation between observed performance is possiblein forming best practice comparison groups, and therefore avoids possibly inappro-priate extrapolation of performance. The extent to which the analyst is interested inscale economies depends on the purpose of the analysis. For example, from a socialperspective, the interest is likely to be in productivity regardless of the scale ofoperations, so the constant returns assumption is most appropriate. However, from amanagerial perspective, there is likely to be considerable interest in the extent towhich the scale of operations affects productivity, so a variable returns assumptionmay be preferred.

The variable returns to scale assumption has the effect of restricting the feasibleregion in (1), and therefore may result in an increase in estimated efficiency. However,given the constant returns production process used in this study, the variable returnsto scale assumption is not appropriate, and so its use may lead to an unnecessarilyrestrictive region of search for efficient DMUs. The purpose of this section is there-fore to explore the implications of wrongly abandoning the constant returns to scaleassumption by invoking constraint (4).

The effect of invoking the variable returns constraint is illustrated in figure 3,for the case of a production process with just one input and one output. Under con-stant returns, the DEA frontier is estimated as the straight line OP*, passing throughthe single efficient DMU P* . The variable returns frontier is P1P

*P2P3, which passesthrough the DMUs P1, P2 and P3, as well as P* . Under variable returns, DMUs usingthe lowest level of any input or producing the highest level of any output must bedeemed DEA efficient, regardless of their intrinsic efficiency. For such DMUs, theonly feasible combination of λj’s that satisfies (4) as well as the relevant constraintin (1) is that which places a weight of one on the DMU of interest and zero on allother DMUs.

The variable returns constraint is likely to have the greatest effect on efficiencyestimates when sample sizes are small, as with larger sample sizes there is a greaterprobability of being able to form a comparison group with weights which conform to (4)and which exhibits performance which is close to the efficiency of the unconstrainedcomparison group. This intuition is confirmed by table 5, which presents the increasesin efficiency brought about by the variable returns to scale assumption. Clearly none


of the previously efficient DMUs is affected by the additional constraint. However,the new assumption increases the estimated efficiency of almost all the DMUs deemedinefficient using the correct model. The table shows that for all five models consid-ered, the overestimate in DEA efficiency brought about by invoking the constraintdiminishes as sample size increases. This effect is most marked for the simplest twoinput model, for which the extent of the overestimate decreases from 21% to 9% assample size increases. For the most complex six input model, the overestimate is moremodest (15%) for small samples, and decreases by a smaller margin to 9% for largesamples.

Figure 3. Impact on production frontier of imposing variable returns to scale.

Table 5

Increases in efficiency arising from invoking variable returns to scale.


10 20 40 80

Two Average efficiency 0.7177 0.6308 0.5712 0.5298inputs Overestimate (%) 21.5 16.8 12.5 8.8

Three Average efficiency 0.7664 0.6853 0.6198 0.5728inputs Overestimate (%) 18.6 15.3 12.7 9.8

Four Average efficiency 0.8010 0.7203 0.6543 0.6024inputs Overestimate (%) 17.6 15.2 12.5 10.0

Five Average efficiency 0.8294 0.7518 0.6867 0.6347inputs Overestimate (%) 16.0 14.2 11.7 9.9

Six Average efficiency 0.8377 0.7614 0.6983 0.6456inputs Overestimate (%) 14.8 12.9 10.9 8.9


5. Robustness of results

In prosecuting this study, although an attempt has been made to make the modelas uncontroversial as possible, many arbitrary assumptions have perforce been adopted.Some of the assumptions – such as the functional form of the production process andthe distribution of inefficiencies – are structural, and are required to generate a mean-ingful set of observations. However, other assumptions are less intrinsic to the studymethodology, and alternatives can be tested. This section discusses them.

The presentation in section 4 has already indicated the impact of altering twokey elements: the size of samples used in DEA, and the number of inputs in the pro-duction process. Clearly it is possible to increase sample sizes indefinitely. However,the size of samples used here is typical of the sizes available in empirical studies, andin any case table 1 suggests that the expected marginal impact of including an extraobservation in the DEA model declines rapidly as sample size increases. As a result,we feel the range of sample sizes considered here is adequate.

Similarly, it is possible to posit alternative mixes of inputs and outputs. A DEAstudy is unlikely to use more than six inputs. However, one of the key advantages ofthe technique over standard econometric methods is that it permits the use of multipleoutputs. This study has not considered the possibility explicitly, being confined toconsideration of a single output. However, note that input and output constraints inmodel (1) are very similar. Indeed, in the context of this model, exogenously fixedoutputs can be considered as negative inputs [4]. The biggest determinant of theeffects described here is therefore likely to be the total number of inputs and outputs.The number of variables considered here varies between three and seven, and as aresult we effectively cover a wide spectrum of possible models.

There are however two areas where further robustness analysis may be valuable:the covariance structure of inputs, and the importance of inputs in the productionprocess. The rest of this section therefore assesses the sensitivity of the results to thesetwo considerations.

5.1. Covariance between inputs

So far, the analysis has assumed covariances between all inputs are equal to0.8. This assumption, although reflecting the sort of situation found in practice, is ofcourse arbitrary. Furthermore, it may be less appropriate in more complex, multipleoutput production processes. In order to test the importance of the covariancestructure, we examine here the impact of adopting a radically different assumption:that of zero correlation between inputs. All the experiments in section 4 are thereforererun using an identical methodology, with the exception of the revised covarianceassumption. The salient features are summarized in table 6 for the four input model,which is representative of all models. Full results are available from the author.

The results indicate that the reduced correlation between inputs tends to exag-gerate the phenomena reported in section 4. Thus, for example, overestimates of


Table 6

Impact of zero correlation between inputs on study findings (four-input model).

CovarianceSample size

between inputs10 20 40 80

Correct DEA model: 0.8 51.3 38.9 29.2 21.6overestimate of efficiency 0.0 78.5 63.7 50.6 39.4

Omission of input: 0.8 – 7.8 – 9.2 – 10.5 – 11.8underestimate of efficiency 0.0 – 14.5 – 18.5 – 22.2 – 25.5

Extraneous variable: 0.8 3.3 3.6 3.4 3.1overestimate of efficiency 0.0 3.7 4.4 4.9 5.0

Variable returns to scale: 0.8 17.6 15.2 12.5 10.0overestimate of efficiency 0.0 15.3 16.8 16.4 14.9

efficiency yielded by DEA using the correctly specified model are substantially largerthan those reported in table 1. High covariance between inputs therefore allows DEAto yield better estimates of efficiency.

Similarly, when an input is omitted, the reductions in efficiency that result aremore than those arising when there is positive correlation between inputs. For example,when a sample size of 40 is used, average DEA efficiencies are underestimated by22.2%, almost double the 10.5% underestimate found with positive covariance (table 2).This result accords with intuition. With strong positive correlation, the inputs thatremain in the model contain some information about the omitted input, and the modelis therefore less sensitive to its exclusion. However, it should be emphasized that– even with the strong correlation assumed in section 4 – omission of a relevant vari-able still leads to substantial underestimates of DEA efficiency with simple models.

The inclusion of an extraneous variable has only a modest impact on DEA effi-ciency rankings when inputs are uncorrelated. Thus, the results described in section 4are sustained. There is a substantial asymmetry between the large bias introduced bythe exclusion of a relevant variable and the relatively small bias caused by inclusionof an extraneous variable. Apart from when sample sizes are very small, the variablereturns to scale misspecification is more serious with uncorrelated inputs.

5.2. Importance of inputs

The parameter α plays a crucial role in the production model used here. Differ-entiating the production function implicit in (2) with respect to xi yields the familiarresult

α ii

i

xy

yx

= ∂∂

.


That is, αi indicates the elasticity of production with respect to the input xi underefficient production. In intuitive terms, therefore, other things being equal theparameter indicates the importance of input xi in the production process. All the resultsto date have been based on the m input model with αi = 1ym. In practical applica-tions, there are likely to be considerable variations in the relative importance of inputsin the production process. It is therefore important to test the robustness of the resultsto the assumption of equal importance. Accordingly, the analysis was repeated withthe same simulated input and inefficiency data, but with revised parameters α1 = 9y5mand α2 = 1y5m. All other αi are unchanged at 1ym. Thus, a one percent increase in x1

is assumed to yield percentage output growth nine times more than that obtained froman equivalent increase in x2, and 80% more than that from all other xi .

Table 7 reports the effect of omitting variables associated with different valuesof αi from the DEA models using sample sizes of 40. The first two lines show theefficiency estimates yielded by the correct DEA models. They are on average veryslightly closer to the true efficiencies than those found in the original models (table 1).The subsequent lines show a considerable asymmetry in the impact of the omission

Table 7

Actual and DEA average efficiencies with α1 = 9y5m, α2 = 1y5m and αm = 1ym(sample sizes 40).

Number of inputs in model

2 3 4 5 6

Average DEA efficiency 0.4970 0.5385 0.5723 0.6079 0.6235(correct model)

Overestimate of true 10.1 19.5 27.1 33.8 37.9efficiency (%)

DEA efficiency with 0.2043 0.3709 0.4668 0.5375 0.5705omitted input x1

Underestimate of DEA – 58.9 – 31.1 – 18.4 – 11.6 – 8.5efficiency (%)

DEA efficiency with 0.4483 0.5045 0.5456 0.5849 0.6050omitted input x2

Underestimate of DEA – 9.8 – 6.3 – 4.7 – 3.8 – 3.0efficiency (%)

DEA efficiency with NA 0.4471 0.5099 0.5637 0.5909omitted input xm

Underestimate of DEA NA – 17.0 – 10.9 – 7.3 – 5.2efficiency (%)


of a variable. Omission of the “important” variable x1 renders the DEA estimatesalmost meaningless with simpler models, with average efficiencies falling to 0.20– an underestimate of 59% – in the two input case. However, as model complexityincreases, the bias reduces to 9%. Omission of the relatively unimportant variablex2 leads to much smaller efficiency underestimates. However, it is noteworthy that,notwithstanding the very small value of α2, the efficiency underestimates introducedby its omission are substantially larger than those caused by including an extraneousvariable in the model (table 3). This confirms the view that exclusion of relevantvariables is likely to be more damaging to DEA models than inclusion of irrelevantvariables. The omission of the remaining variable xm leads to intermediate levels ofbias.

6. Discussion

Using a well-known productivity model, this paper has sought to assess theaccuracy of efficiency estimates yielded by DEA. The paper has examined threephenomena relevant to building satisfactory DEA models: omission of salient variables;inclusion of extraneous variables; and the returns to scale assumption. Of course, theprecise magnitude of the results is often highly dependent on the detailed assump-tions incorporated into the model. In particular, the findings must be viewed in thelight of the assumption that all inputs are drawn from normal distributions. Section 5seeks to assess the robustness of the results to some of the other assumptions adoptedin this paper.

It has been shown that – if the model is simple and the DEA representation wellspecified – DEA yields accurate estimates of true efficiency for many DMUs, andthat the main inaccuracies arise for DMUs that DEA judges to be 100% efficient.Particular care should therefore be taken in interpreting DEA results for DMUsdeemed efficient. This distortion is reduced as sample size increases. However, as thecomplexity of the model increases, the inaccuracies in efficiency estimates yielded byDEA rapidly increase. In this paper, we have shown that, with only six inputs and oneoutput, DEA overestimates true efficiency by almost one third, even with large samplesizes. Therefore, users should be aware that, for complex production processes, DEAmay yield very conservative estimates of potential efficiency savings.

The robustness analysis has examined the sensitivity of the results to samplesize; correlation between inputs; variations in the importance of inputs; and variationsin the number of inputs. The omission of salient variables has been shown to be veryimportant for a simple production process. However, with six inputs the omission ofa variable led to less serious underestimates of DEA efficiency. The inclusion of anextraneous variable appears to have only modest implications for average efficiencyestimates, regardless of the complexity of the model. The inappropriate use of avariable returns to scale assumption is particularly damaging when sample size issmall.


From the model builder’s perspective, therefore, when the importance of vari-ables is in doubt, it seems appropriate to include variables which might be important,especially if the model being used includes only a small number of variables. Thecosts incurred by including a redundant variable are likely in most circumstances tobe lower than the costs of excluding a relevant variable. The results in section 5suggest that this consideration is likely to be particularly important when there is lowcorrelation between inputs. This study therefore indicates that the parsimony criterionadopted for econometric models is less relevant for DEA, particularly when the modelis simple or the sample size small.

Similarly, the decision as to whether or not to invoke the variable returns to scalecriterion is most critical when the model is simple and sample size is small. In theabsence of information about the returns to scale implicit in the production process,use of the variable returns criterion is often justified on the grounds that it offersconservative estimates of achievable productivity improvements. This study has shownthat, even with large sample sizes, average overestimates of DEA efficiency of theorder of 10% arise if variable returns are incorrectly assumed when returns are inpractice independent of scale.

Throughout, it is important to bear in mind that we have concentrated on averageeffects of misspecification on efficiency estimates. The results are therefore mostuseful for judging the accuracy with which DEA estimates the aggregate levelof inefficiency within the entire sample – representing an industry or governmentprogramme [7]. However, the impact on a particular DMU might be considerable,even when the average effect is small. While exigences of space preclude detailedanalysis, we have sought to indicate in general that particular forms of misspecifi-cation might tend to favour specific types of DMU to the disadvantage of others. Thus,for example, omission of a salient input, other things being equal, tends to result inbigger reductions in efficiency estimates in DMUs employing low levels of that input.This particular misspecification therefore will lead to such DMUs being set the tough-est efficiency targets. Similarly, the incorrect use of the variable returns criterion islikely to favour DMUs operating at unusually large or small scales. This managerialbias (in contrast to the programme bias emphasized here) clearly merits furtherresearch. The correlation analysis reported in table 4 suggests that the magnitude ofbiases in the ranking of DMUs is likely to mirror very closely the magnitude of biasesin average efficiency levels reported here.

The principal aim of this paper has been to initiate a framework for developinga modelling strategy for the DEA user. Because of the lack of model selection devices,misspecification will always be a central concern, and this paper has highlighted aconundrum. Using realistic sample sizes, DEA yields the most accurate estimates oftrue efficiency for simple production processes. However, it is models of preciselysuch processes which are most sensitive to misspecification. The DEA user musttherefore be constantly vigilant. The conclusions from this paper are, within reason,(a) to examine with some care the performance of DMUs deemed to be efficient; (b) to


err on the side of proliferation of variables in the specification of a DEA model; (c) tobe cautious about invoking variable returns to scale when sample sizes are small; and(d) to be especially cautious about results from models exhibiting low correlationbetween inputs.

Acknowledgements

This paper was presented at the TIMSyORSA Joint National Meeting in Boston,April 1994. Thanks are due to my colleagues Chris Orme and Les Godfrey, to theparticipants at the Boston meeting, and to the editors and referees for their helpfulcomments.

References

[1] T. Ahn and L.M. Seiford, Sensitivity of DEA to models and variable sets in a hypothesis test set-ting: The efficiency of university operations, in: Creative and Innovative Approaches to the Scienceof Management, Y. Ijiri, ed., Quorum Books, Westport, 1993.

[2] A.I. Ali and L.M. Seiford, The mathematical programming approach to efficiency analysis, in: TheMeasurement of Productive Efficiency, H.O. Fried, C.A.K. Lovell and S.S. Schmidt, eds., OxfordUniversity Press, Oxford, 1993.

[3] R.D. Banker, Econometric estimation and data envelopment analysis, Research in Government andNonprofit Accounting 5(1989)231–243.

[4] R.D. Banker and R.C. Morey, Efficiency analysis for exogenously fixed inputs and outputs, Opera-tions Research 34(1986)513–521.

[5] R.D. Banker, A. Charnes and W.W. Cooper, Some models for estimating technical and scaleefficiencies in Data Envelopment Analysis, Management Science 30(1986)1078–1092.

[6] R.D. Banker, A. Charnes, W.W. Cooper and A. Maindiratta, A comparison of DEA and translogestimates of production frontiers using simulated observations from a known technology in: Appli-cations of Modern Production Theory: Efficiency and Productivity, A. Dogramaci and R. Färe, eds.,Kluwer, Boston, 1988.

[7] A. Charnes and W.W. Cooper, Auditing and accounting for program efficiency and managementefficiency in not-for-profit entities, Accounting, Organizations and Society 5(1980)87–107.

[8] A. Charnes, W.W. Cooper, D. Divine, T.W. Ruefli and D. Thomas, Comparisons of DEA andexisting ratio and regression systems for effecting efficiency evaluations of regulated electric co-operatives in Texas, Research in Government and Nonprofit Accounting 5(1989)187–210.

[9] A. Charnes, W.W. Cooper, A.Y Lewin, R.C. Morey and J. Rousseau, Sensitivity and stability analy-sis in DEA, Annals of Operations Research 2(1985)139–156.

[10] A. Charnes, W.W. Cooper and E. Rhodes, Measuring the efficiency of decision making units,European Journal of Operational Research 2(1978)429 – 444.

[11] A. Charnes, W.W. Cooper and E. Rhodes, Short communication: measuring the efficiency of deci-sion making units, European Journal of Operational Research 3(1979)339.

[12] A. Charnes, W.W. Cooper and R.M. Thrall, A structure for classifying and characterizing efficiencyand inefficiency in Data Envelopment Analysis, Journal of Productivity Analysis 2(1991))197–237.

[13] J. Doyle and R. Green, Efficiency and cross-efficiency in DEA: Derivations, meanings and uses,Journal of the Operational Research Society 45(1994)567–578.

[14] M.K. Epstein and J.C. Henderson, Data Envelopment Analysis for managerial control and diagno-sis, Decision Sciences 20(1989)90–119.


[15] L.G. Godfrey, Misspecification Tests in Econometrics, Cambridge University Press, Cambridge,1988.

[16] D.F. Hendry, PC-GIVE: An Interactive Econometric Modelling System, Institute of Economics andStatistics, University of Oxford, Oxford, 1989.

[17] T.R. Nunamaker, Using Data Envelopment Analysis to measure the efficiency of non-profitorganizations: A critical evaluation, Managerial and Business Economics 6(1985)50–58.

[18] A. Pagan, Three econometric methodologies: A critical appraisal, Journal of Economic Surveys1(1987)3 –24.

[19] L.M. Seiford and R.M. Thrall, Recent developments in DEA, Journal of Econometrics 46(1990)7–38.

[20] J.K. Sengupta, Data Envelopment Analysis for efficiency measurement in the stochastic case,Computers and Operations Research 14(1987)117–129.

[21] R.M. Thrall, Classification transitions under expansion of inputs and outputs in DEA, Managerialand Decision Economics 10(1989)159–162.

[22] V. Valdmanis, Sensitivity analysis for DEA models. An empirical example using public vs. NFPhospitals, Journal of Public Economics 48(1992)185–205.


Documents

Model misspecification in Data Envelopment Analysis