Estimating Effort and Cost in Software Projects ISBSG …s3.amazonaws.com/publicationslist.org/data/p.bourque/ref-785/979.pdf · Teams communicationTechnology changes ... Logistics

Estimating Effort and Cost in Software Projects

−ISBSG A Multi-Organizational Project Data Repository for Project Estimation

And Benchmarking

BriefingCentre for Software Process Technologies

University of UlsterNorthern Ireland

18 September 2003

Pierre BourqueÉcole de technologie supérieure

Montreal, Canada

Centre for Software Process Technologies

2

Benchmarking the NI software industry

Concern measures for Engineering Aspects

109106

9993 91

8885 84 83 81 79

7469

65

49

0

20

40

60

80

100

120

Estim

ating T

asks

Risk m

anag

emen

t

Produ

ctivity

Projec

t Trac

king

Work pr

actise

cons

istenc

y

S/w re

work

Proce

ss Adh

erenc

e

Projec

t Plan

ning

Ensu

ring q

uality

Develo

ping R

equir

emen

ts

Manag

ing R

equir

emen

ts

Maintai

ning s

/w

Team

s com

munica

tion

Tech

nolog

y cha

nges

Manag

ing fau

lts

Engineering Aspect

Cu

mu

lati

ve C

on

cern

Val

ue


3

Ice Bag

ISBSG


4

École de technologie supérieure -ETS

• An Engineering University– Within the network of the Université du Québec

• In + 10 cities across Québec

• Engineering Specialties:– Aeronautical Engineering – Automated Manufacturing – Construction Engineering– Electrical Engineering– Mechanical Engineering – Software Engineering– Telecommunications + Information Technologies


5

ETS Software Engineering Lab. Strengths

• Recognized world-wide in software engineering for:– Building consensus in software engineering– Leadership of world-wide initiatives– Strong applied research focus

• www.lrgl.uqam.ca


6

Software Engineering Management

• Sample of R & D topics:– Guide to the Software Engineering Body of Knowledge

(SWEBOK)– Second generation of functional size method: COSMIC – Estimation models– Quality Engineering– Risk of measurement programs– Metrology


7

Agenda

• Principles of Credible Estimation• Overview of Software Functional Sizing• Overview of ISBSG• Overview of the Repository• An example of using ISBSG for Duration

Estimation• Conclusion


8

How do you build your estimates?

• How do you build estimates in your organization?• What are the inputs to your estimates?• Is your estimation process documented?• Do you collect and use historical data?• Do you collect data that is never used?• What is the reliability of this data?• Do you track your estimates?• How do you size the amount of work to be done?• Are the quality of your estimates very much based on the

competency of a few key people in your organization?


9

What is software engineering ?

• IEEE 610.12:(1) The application of a systematic, disciplined,

quantifiable approach to the development, operation, and maintenance of software; that is, the application of engineering to software.

(2) The study of approaches as in (1).


10

Many Commercial Estimation Tools on the Market

• Many well-known commercial tools:– Checkpoint– Project Workbench– PQM Plus– SLIM– …

• See O. Mendes, A. Abran and P. Bourque. Function Point Tool Market Survey. Université du Québec à Montréal,1996 Available at www.lrgl.uqam.ca/publications/pdf/204.pdf


11

What is the result?

¤ Lots of numbers, graphs, nice slide presentation,

¤ Industry data often not verifiable,¤ “Black box” approach,¤ Bottom line: everybody sees what they

want to see...


12

Mixed PUBLISHED Results...

• Panoply of software cost models• Several studies have been conducted to

assess the accuracy of these models on various databases

• However, no study has proven the superiority of any models excepted for limited applications

• Often small data samples


13

Underlying Principles of CREDIBLE Estimation - 1

As Defined in Park et al. (94):« Estimates are made by people, not

by models. They require reasoned judgments and commitments to

organizational goals that cannot be delegated to any automated

process ».


14

Underlying Principles of CREDIBLE Estimation-2

« All estimates are based on comparisons. When people estimate, they evaluate how something is like, and how

something is unlike, things that they or others have seen before ».


15

Underlying Principles of CREDIBLE Estimation-3« Before people can estimate, they must acquire knowledge. They must

collect and quantify information from other projects, so that they can place their comparative evaluations on demonstrably sound footings ».


16

These Principles Imply:

• To be credible, an estimation process must inherently be white-box

• Software project estimation which has been plaguing the industry for years can only be solved through a cooperative data collection effort

• Much research still has to be done


17

Agenda




18

Size of what Size of what ……

HOW BIGHOW BIGIS IT ?IS IT ?

Software sizeSoftware sizethe size of the the size of the requirements (functions) requirements (functions) or of the deliverables or of the deliverables (modules, lines of code)(modules, lines of code)

Project SizeProject SizeThe total effort, The total effort, estimated or actual in estimated or actual in workwork--hours or staffhours or staff--monthsmonths

Con

text

...C

onte

xt...


19

Software size measurementSoftware size measurement

HOW BIGHOW BIGIS IT ?IS IT ?

MmmMmm…… so many programs, so many so many programs, so many lines of code...lines of code...

MmmMmm…… so much functionality so much functionality delivered to the users...delivered to the users...

• Meaningful to the technical staff,Meaningful to the technical staff,•• Meaningless to management,Meaningless to management,•• Poor portability,Poor portability,•• Only known precisely when too late to useOnly known precisely when too late to use

• Meaningful to management,Meaningful to management,•• Meaningful to technical staff,Meaningful to technical staff,•• Portable,Portable,•• Can be measured early on,Can be measured early on,•• Must be independent from effort,Must be independent from effort,

method or technologymethod or technology

TE

CH

NIC

AL

TE

CH

NIC

AL

FUN

CT

ION

AL

FUN

CT

ION

AL

Ø ISO/IEC/JTC1/SC7 Standard #14143ISO/IEC/JTC1/SC7 Standard #14143definitiondefinition::

““ Functional Size : A size of software derivedFunctional Size : A size of software derivedby quantifying theby quantifying the functional userfunctional userrequirementsrequirements””

The The ‘‘Functional SizeFunctional Size’’ of softwareof software


21

An analogy...An analogy...

2000 2000 sqsq. ft.. ft.

4000 4000 sqsq. ft.. ft.

SoftwareFunctionality

SoftwareFunctionality

500 500 cfsucfsu 1000 1000 cfsucfsu


221980 1985 1990 1995 2000

Allan Albrecht

FPA

IFPUG 4.0

IFPUG 4.1

MkIIFPA

MkIIFPA 1.3

Full FP’s V.1

3-D FP’s

Feature Points

ISO ‘FSM’Standard

COSMIC COSMIC FFP V. 2FFP V. 2

Very Significant Amount of Work Very Significant Amount of Work on Software Functional Size?on Software Functional Size?””


23

Usages of Software Sizing

• Estimation• Benchmarking• Productivity Trend Analysis• Contract Payment Mechanisms

– Development– Corrective Maintenance and Support

• Quality Tracking


24

Productivity Trend AnalysisFP Unit Costs

0

10

20

30

40

Eff

ort

1990 1991 1992 1993 1994

Year 0

2

4

6

8

10

12

Ext

end

ed F

P

1990 1991 1992 1993 1994

Year

0

2

4

6

8

10

12

Ho

urs

/Ext

end

ed F

P

1990 1991 1992 1993 1994

Year


25

Agenda




26

ISBSG Mission

• “To help improve the management of IT resources by both business and government through the provision and exploitation of a public repository of software engineering knowledge which is standardized, verified, recent and representative of current technologies.”


27

ISBSG The 1st. Phase... Australian Software Metrics Association,

(ASMA)• Motivation

– practitioners • wanting control• seeking best practices

• Definition of Productivity– Project Delivery Rate = “Hours per Function

Point”


28

ISBSG The 2nd. Phase ...the ASMA Work• First Release of data (24 projects) in

December 1992• Revised standard & collection package

several times• 6 Releases of Data up to June 1994


29

ISBSG The 3rd. Phase

Became the…International Software

Benchmarking Standards Group ISBSG

www.isbsg.org


30

International Membership

Current membership:• Australia, Finland, Germany• India, Italy, Japan, Netherlands,• Spain, Switzerland, • United Kingdom, USA


31

Principles of ISBSG

• Principles of Evolution– independence– data integrity– anonymity– practitioner driven– practitioner accessible


32

Initial Principles of Evolution

• “ Any project team anywhere in the world may, for insignificant cost, benchmark themselves against world’s best”


33

ISBSG Existing Standard

• Collection Philosophy• complete in 30 mins• meaningful, available, objective

• Project Attributes– Functional Size– Work Effort, Team Size, language– Cost – Quality......Etc...........


34

Collection Process

• Security Procedures• Validation Procedures• Repository entry• Project Benchmark Report• Rating comments• Re-submission• Re-rating and project update


35

Summary - ISBSG Strengths

• Not profit motivated• Provides Project Benchmarking• Allows direct access to the data• Allows networking• Broad representation of IT

– technologies, organisation types, geography

• International based standard


36

Agenda




37

ISBSG Release 8

• Demonstration of ISBSG Release 8 Data Set


38

Age of Projects(Data from Release 7)

REPOSITORY PROJECTS

0

50

100

150

200

250

YEAR

PR

OJE

CT

Syear notknown1989

1990

1991

1992

1993

1994

1995

1996

1997

1998

1999

2000

2001


39

Projects by Country

0 50 100 150 200 250 300

Other

Brazil

France

India

United Kingdom

Canada

Netherlands

United States

Japan

Australia

Number of projects


40

Projects by Organization Type

0 50 100 150 200 250 300 350 400 450 500

Other

Engineering

Voice provisioning

Energy, oil, mining

Billing

Ordering

Computers, ISPWholesale & Retail Trade

Electricity, Gas, Water

Banking

Manufacturing

Insurance

Financial, Property & Business Services

Government, Public Administration

Communication

Number of projects


41

Projects by Business Area

0 50 100 150 200 250 300 350

Other

LogisticsResearch & Development

PersonnelLegal

InventorySales & Marketing

EngineeringAccounting

ManufacturingFinancial (excluding Banking)

BankingInsurance, actuarial

Telecommunications

Number of projects


42

Types of Projects

0 200 400 600 800 1000 1200

Re-development

New development

Enhancement

Number of projects


43

Application Type

0 100 200 300 400 500 600

Other

Knowledge Based System

Voice provisioning

Executive Information System

Decision Support System

Process control, sensor control, real time

Electronic Data Interchange

Ordering

Billing

Network Management

Office Information System

Management Information System

Transaction/Production System

Number of projects


44

Use of Development Techniques

0 50 100 150 200 250 300 350 400 450

Standards (ISO 9000; CMM)

Timeboxing

Regression testing

Multifunction teams

Prototyping

Object Oriented Design

Object Oriented Analysis

Rapid App'n Development

Joint App'n Development

Event modelling

Business area modelling

Process modelling

Data modelling

Number of projects


45

The ISBSG Repository -Positioning

• Probably represents top 20% of industry• Primarily MIS Applications


46

Agenda




47

Strategic Importanceof Time-to-Market

• Project manager’s dream:– Complete and stable product requirements– High quality– Low costs– Short time-to-market

• Time to market or project duration is often the hardest one to pin down

• A variation of the other three have a determining effect on it


48

Adopted Viewpointin this Research

ProductRequirements Product Size

ProjectEffort

ProjectDuration

Adopted Viewpoint


49

2. Selecting a Data Sample

2.1 ISBSG release 4 (1997)

2.2 Basic selection criteria

2.3 Distribution analysis

• Effort

• Duration

• Summary


50

2.1 ISBSG Release 4

• Release 4 (1997) contains 396 completed projects – Contribution from 13 countries– 2/3 new development, 1/3 enhancements & re-

development– 34% Txn proc., 38% MIS, 14% office information– 3/4 developped in-house for internal use– 67% Mainframe– 46% 3GL, 38% 4GL


51

2.2 Basic Selection Criteria

• No reasonable doubts on data validity according to ISBSG screening

• Known effort, known duration and known platform

ä 312 projects satisfied all criteria

Duration (D) in calendarmonths

Effort (E) in person-hours

Number of observations (n) 312 312

Minimum value 1,0 10

Maximum value 78,0 106 480

Mean value 10,5 5 933

Standard deviation 9,0 12 169

Median 8,0 2 228

D range: 1 to 78 monthsE range: 0,1 to 761 person-months


52

2.3 Basic Criteria

Scatter plot of effort vs. duration (n=312)

person-hours

months

DU

RA

TIO

N

0

10

20

30

40

50

60

70

80

0 20000 40000 60000 80000 100000

EFFORT


53

2.3 Distribution Analysis - Effort

No transform

Log transformed

Statistic Value Significance (α <= 0,05)

Skewness (√b1) 4,87 Hypothesis of normality rejected

Kurtosis (b2) 33,69 Hypothesis of normality rejected

Combined (K2) 344,25 Hypothesis of normality rejected


Skewness (√b1) 0,05 Hypothesis of normality NOT rejected

Kurtosis (b2) 3,26 Hypothesis of normality NOT rejected

Combined (K2) 1,28 Hypothesis of normality NOT rejected

100

200

300

010000 30000 50000 70000 90000 110000

25

50

75

1.0 2.0 3.0 4.0 5.0


54

2.3 Distribution Analysis -Duration

No transform

Log transformed


Skewness (√b1) 2,88 Hypothesis of normality rejected

Kurtosis (b2) 15,78 Hypothesis of normality rejected

Combined (K2) 222,49 Hypothesis of normality rejected


Skewness (√b1) 0,07 Hypothesis of normality NOT rejected

Kurtosis (b2) 3,37 Hypothesis of normality NOT rejected

Combined (K2) 2,17 Hypothesis of normality NOT rejected

50

100

150

0 10 20 30 40 50 60 70 80

20

40

60

0 1 2


55

2.3 Distribution Analysis Summary

• Skewness due to the natural distribution of projects

• Normal distribution cannot be assumed without log transformation

• Log transformed data selected for modeling purposes.


56

3. Deriving Models

3.1 Correlation analysis

3.2 Regression analysis:

Selected results

Residual analysis

The empirical models

Summary


57

3.1 Correlation Analysis(MF Platform)

Scatter plot of Log(effort) vs. Log(duration), n=208

• Pearson correlation coef. (r): 0,72

• Significant at 0,05 confidence level

• Linear model preferred

LOG

_DU

R

0

1

1,0 2,0 3,0 4,0 5,0LOG_EFF


58

3.2 Regression Analysis

• Regression hypotheses:

– linear relation judged adequate

– residuals are randomly distributed

– residuals independent from independent

variable

– variance of residuals is constant


59

3.2 Regression Analysis -Selected Results

(MF Platform)

Selected results Value

Sample size (n) 208

R2 0,522

F(1,207) 224,865

Prob. > F 0,0001

Log(E) coefficient 0,366

Standard error of Log(E) 0,024

Constant -0,339

ä Independent variable: Log(Effort)

ä Dependent variable: Log(Duration)

ä Linear regression model


60

3.2 Regression Analysis -Residual Analysis

(MF Platform)

• Residuals are randomly distributed

• Residuals are independent of Log(E)Res

idua

l

-1

0

1

1 2 3 4 5LOG_EFF


61

3.2 Regression Analysis -Residual Analysis

(MF Platform)

Variance of residuals is

constant over the range of the

dependent variable Log(D)

Res

idua

l

0

0 1LOG_DUR Predicted


62

3.2 Regression Analysis -The Empirical Model

(MF Platform)ä Directly from regression results:

ä Converted to the usual format:

Log(D) = (0,366 * Log(E)) - 0,339 (E in person-hours)

D = 0,458 * E 0,366 (E in person-hours)


63

3.2 Regression Analysis - The Empirical Models(MR and PC Platform)

ä MR platform model (n = 65):

ä PC platform model (n = 39):




64

3.2 Regression Analysis - Summary

0

1

2

1.0 2.0 3.0 4.0 5.0LOG_EFF

PC

MF

MR

MF platform : D = 0,458 * E 0,366

MR platform : D = 0,548 * E 0,360

PC platform : D = 1,936 * E 0,201


65

4. Assessing the Models

Criteria ESCOM‘97

ISBSGr. 3

MFmodel

ISBSGr. 4

MRmodel

ISBSGr. 4

PCmodel

ISBSGr. 4

n 243 208 65 39R2 0,40 0,41 0,49 0,06Rank 3 2 1 4Avg. RE - 0,18 - 0,15 - 0,14 - 0,16Rank 4 2 1 3Avg. MRE 0,48 0,45 0,47 0,48Rank 3 1 2 3Pred (0,25) 38 % 43 % 34 % 41 %Rank 3 1 4 2RMS 7,20 6,78 8,52 5,45Rank 3 2 4 1RMS bar 0,69 0,68 0,68 0,57Rank 3 2 2 1

ä Value of Conte et al. (86) criteria for untransformedestimates:

ä Platform dependent models all show a performance equal or better thanthe ISBSG r.3 model (except for three values),

ä Improvements are small though,

ä In all models, magnitude of criteriaunderline the usefullness of model for “ballpark” estimates only. (ex.: Avg. MRE and Pred(0,25))


66

Agenda




67

Conclusion

• Software sizing is different from estimation• ISBSG data is available and can be analyzed by

everyone.• The steps taken to derive the example model and

the assumptions behind it are known and the accuracy for this sample is published.

• Allows more intelligent tradeoffs and informed choices between various scenarios.


68

Conclusion• Development of demonstrably sound

quantitative models is a difficult and key problem in this industry.

• Can only be solved with an inherently white-box approach.

• Credibility of results depends entirely on the transparency of the method, data, definitions and assumptions that were used to derive this estimate.


69

6. Further Research Topics

ä Turn the problem around

ProductRequirements Product Size

ProjectEffort

ProjectDuration

New Viewpoint


70

Refererences

• S.D. Conte, H.E.Dunsmore, V.Y. Shen, Software engineering metrics and models. Menlo Park: TheBenjamin/Cummings Publishing Company, Inc. 1986.

• O. Mendes, A. Abran and P. Bourque. Function Point Tool Market Survey. Université du Québec àMontréal,1996 Available at http://saturne.info.uqam.ca/Labo_Recherche/Lrgl/publi/treports/mo199701/mo199701.pdf.

• R. E.Park, W. B. Goethert and J.T. Webb. Software Cost and Schedule Estimating: A Process Improvement Initiative. Pittsburgh, PA Software Engineering Institute,1994.

Documents

Estimating Effort and Cost in Software Projects ISBSG …s3.amazonaws.com/publicationslist.org/data/p.bourque/ref-785/979.pdf · Teams communicationTechnology changes ... Logistics