Revisiting Software Development Effort Estimation Based on Early Phase Development Activities

MSR 2013Masateru Tsunoda, Koji Toda, Kyohei Fushida, Yasutaka Kamei, Meiyappan Nagappan, Naoyasu Ubayashi

(Kinki University, Japan; Fukuoka Institute of Technology, Japan;NTT DATA Corporation, Japan; Kyushu University, Japan;Queen's University, Canada)

2

Effort is estimated using software size Total effort is estimated based on software size to

decide staffing and schedule. High accuracy is needed to avoid project failure.

Linear regression is one of common methods to build the model [1]. Size is settled by a method such as function point analysis.

[1] L. Briand, T. Langley, and I. Wieczorek, “A replicated assessment and comparison of common software cost modeling techniques,” In Proc. of International Conference on Software Engineering (ICSE), pp. 377–386, Limerick, Ireland, June 2000.

effort (estimated) = 1.5 + 0.069×size

Estimated effort = 4 person-month

2 developers 2 months×

3

Estimation using early phase activity Practitioners use the ratio of early phase activities to

the whole phase.

Which shows higher estimation accuracy?

early phase efforttotal effort (estimated)

Ratio of effort before design phase (estimation timing) is 33% (on avg.).

Req. analysis Design TestingCodingPlanning

Req. analysis Design TestingCodingPlanning

effort (estimated) = 1.5 + 0.069×sizeeffort (estimated) = 3×early phase effort

effort (estimated) = 3×early phase effort

Estimation timing

4

What should be clarified? RQ1: When a model using software size or early

phase effort is built, which shows higher accuracy? Collecting data requires effort, so some organizations do

not have detailed data.

RQ2: When other variables are added to the models on RQ1, which model shows higher accuracy?

Softwaresize Total

effortModel Early phase

effort Totaleffort

Model

ProgrammingLanguage

Add (RQ2)Programming

Language

Add (RQ2)

5

What should be clarified? (contd.) RQ3: When both software size and early phase effort

are used, is estimation accuracy improved?Does multicollinearity arise by using them?

Softwaresize Total

effortModel

Early phaseeffort

6

Built models using ISBSG dataset 118 projects (data point) in ISBSG dataset

Collected from organizations in 20 countries by ISBSG.

Definition of early phase effort Planning effort Planning-and-analysis effort

Built model FP (software size): baseline planning effort planning-and-analysis effort planning effort, FP planning-and-analysis effort, FP

Req. analysisPlanning

Req. analysisPlanning

Softwaresize

Early phaseeffortEarly

phaseeffort

Softwaresize

7

How to compare estimation accuracy Evaluated accuracies by differences in balanced

relative error (BRE) from the baseline (FP model).

effort = a x FP (baseline) average BRE = 100%effort = b x planning effort average BRE = 50%

Difference of average BRE = 100% - 50% = 50%improved

Estimated effort 50 hourActual effort 100 hour

BRE = |100-50| / 50 = 100%

Example of BRE

8

Early phase effort improves accuracy Accuracy of a model using early phase effort is higher

than software size. Built models without other variables.

Multicollinearity did not arise when using both early phase effort and FP.

0% 50% 100%

Planning-and-analysiseffort, FP

Planning effort, FP

Planning-and-analysiseffort

Planning effort Median BREAverage BRE

Size (FP) and early phase effort > Size(Ans. to RQ3: When other variables are not used)

Difference

9

Other variables did not work well Adding other variables improved accuracy of FP

model. But it did not improve accuracy of other models.

Other variables: development type, platform type, language type

Set baseline as FP with the variables, and compared other models without the variables.

effort = a x FP+ c x platform type + … (baseline)Avg. BRE = 100%

effort = b x planning effort Avg. BRE = 50%

Difference of average BRE = 100% - 50% = 50%improved

10

Accuracy of a model using planning-and-analysis effort is higher, but planning effort is not than software size (Ans. to RQ2).

Using both early phase effort and software size improves accuracy, and multicollinearity does not arise. (Ans. to RQ3)

Early phase effort is still effective

-10% 10% 30% 50% 70%

Planning-and-analysis effort, FP

Planning effort, FP

Planning-and-analysis effort

Planning effort Median BREAverage BRE

Size and early phase effort> Size

Difference

11

How to build estimation model? Preferable to build a model that only uses early

phase effort as an explanatory variable. If an organization does not collect data in detail

It might not be preferable to use variables which we used as additional variables. In organizations that collect other data in detail

Using both early phase effort and software size improves the accuracy without multicollinearity. If software size is settled precisely by a method such as

function point analysis