Software Measurement Now and the Future · ( International Function Point User Group) ... ( COmmon Software Measurement International Consortium) – research, development and trials

1

© Total Metrics

Software MeasurementSoftware MeasurementNow and the FutureNow and the Future

JFPUG Conference Tokyo

April 2000

Pam Morris - Total Metrics

2

© Total Metrics

Pam MorrisPam Morris(BSc. Grad. Dip Computing, Dip. Ed.)(BSc. Grad. Dip Computing, Dip. Ed.)

• Member of the IFPUG CountingPractices Committee since 1992

• Workgroup convenor and projecteditor ISO/IEC 14143 Functional SizeMeasurement Standards

• Executive Member of the AustralianSoftware Metrics Association (ASMA)

• Core project member COSMIC• Chief Executive Officer of Total Metrics

3

© Total Metrics

Aims of this presentationAims of this presentation

• To talk about current status ofSoftware Measurement and its use inAustralia

• To describe the benefits found , issuesexperienced and the needs identified

• To report on a new measurementinitiative - COSMIC , its aims, designs,principles, progress and interimfindings.

4

© Total Metrics

Current Status - SoftwareCurrent Status - SoftwareMeasurement in AustraliaMeasurement in Australia

ØMain areas of Use :– Monitoring Outsourcing

Agreements– Evaluating Software Assets– Estimating and Monitoring

fixed price projects– Process Improvement and

Benchmarking

5

© Total Metrics

Outsourcing AgreementsOutsourcing Agreements

ØMain areas of Use :– Portfolio Assessment– Performance Improvements– Productivity Rates– Estimating Enhancements

6

© Total Metrics

Outsourcing AgreementsOutsourcing Agreements

ØBenefits– Objective quantification of :

• performance targets based onoutputs

• portfolio sizeØ Issues :

– FPA is not a ‘silver bullet’– Focuses on functionality rather

than ‘value’– Need additional metrics

7

© Total Metrics

Software AssetsSoftware Assets

ØMain areas of Use :– Accrual Accounting– Value based on

‘replacement cost’

8

© Total Metrics

Software AssetsSoftware AssetsØBenefits :

– Provides more realisticevaluation of softwareas a ‘capital asset’

Ø Issues :– high cost– need a more rigorous

FPA standard– Need to value

‘knowledge capital’

9

© Total Metrics

Estimating and MonitoringEstimating and MonitoringFixed Price ContractsFixed Price Contracts

ØMain areas of Use :– Estimating resources– Comparison of solutions– Negotiating scope– Dollar cost per unit delivered– Quantifying scope creep and

rework– Evaluating Risk

10

© Total Metrics


ØBenefits :– Quantitative objective

measure of delivered productsize

– Realistic estimates– Quantification of scope creep

and rework– Quantitative Risk evaluation

11

© Total Metrics


Issues :– Impact of productivity factors -

project and product attributes– Which ‘changes’ can be counted

using FPA– Sensitivity of FPA– Different Interpretations of rules– IFPUG not as effective in all

software functional domains

12

© Total Metrics

Process Improvement andProcess Improvement andBenchmarkingBenchmarking

ØMain areas of Use :– Internal comparison– Spice / CMM assessment– External comparison

13

© Total Metrics


ØBenefits:– Quantitative comparison– Capability to set targets and

quantitatively monitorimprovements

– Availability of externalindustry data

14

© Total Metrics


ØIssues :– Resources and cost– Lack of available ‘like’ data– Adhoc approach rather than

‘balanced scorecard’– Lack of available integrated

metrics tools

15

© Total Metrics

Industry NeedsIndustry NeedsØ Wish List:

– Wider range of metrics for specificneeds

– International standardisation ofmetrics

– Better Metrics -– simpler– more accurate– higher consistency– guaranteed repeatability– wider scope

– Automated metrics collection ,analysis and reporting

– Improved availability of industryrelevant data

– Better understanding of metricscapability and measurement concepts

16

© Total Metrics

Current work towards the Future -Current work towards the Future -Functional Size MeasurementFunctional Size Measurement

• ISO ( International Standards Organisation)

– ISO/IEC 14143 Functional SizeMeasurement Standard

• ISBSG ( International Software Benchmarking StandardsGroup)

– accepting FSM results from new methods.

• IFPUG ( International Function Point User Group)

– updating CPM 4.1 - removing AdjustmentFactors

• COSMIC ( COmmon Software MeasurementInternational Consortium)

– research, development and trials of new FSMmethod

17

© Total Metrics


• To talk about current status ofSoftware Measurement and its use inAustralia

• To describe the benefits found , issuesexperienced and the needs identified

• To report on a new measurementinitiative - COSMIC , its aims, designs,principles, progress and interimfindings.

18

© Total Metrics

COSMIC FFP Field TrialsCOSMIC FFP Field TrialsSummary of Aims,Summary of Aims,

Progress and InterimProgress and InterimFindingsFindings

Presented on behalf of the COSMIC CoreTeam *

April 2000

(* Alain Abran, Charles Symons, Jean-Marc Desharnais,Peter Fagg, Pam Morris, Serge Oligny, Jolijn Onvlee,Roberto Meli, Risto Nevalainen, Grant Rule, Denis St

Pierre)

‘COSMIC’ - the COmmon Software MeasurementInternational Consortium

19

© Total Metrics

• To introduce the COSMICproject, its aims and structure

• To describe the current status of theproject and interim findings

• To invite interested organisations toparticipate in the next stage of field-testing of the COSMIC principles

Version 2 April © COSMIC 2000

COSMIC FFP - Aims,COSMIC FFP - Aims,Design Principles andDesign Principles and

ProgressProgress

20

© Total Metrics

Why the focus onWhy the focus onimproving Functional Sizeimproving Functional Size

MeasurementMeasurement(FSM)?(FSM)?

“The ability to measurethe size of software ishugely important forsoftware managementand managing projectsand contracts”


21

© Total Metrics

Industry needs betterIndustry needs bettersoftware sizing methodssoftware sizing methods

Current methods for sizing requirements(i.e. ‘Function Point’ methods) arewidely accepted in the MIS domain butare unsatisfactory as long-termsolutions

– limited acceptance outside the traditionalMIS world

– do not deal satisfactorily with ‘non-functional’ (= technical and quality)requirements

– rules are not always compatible with modernsoftware requirements and developmentmethods and are sometimes subjective ininterpretation Version 2 April © COSMIC 2000

22

© Total Metrics

COSMIC design principles:COSMIC design principles:summarysummary

• COSMIC started November 1998, 10 countriesparticipated in development of the design.

• COSMIC’s basic design principles wereworked out and agreed by the Core Team

• The COSMIC method has achieved a numberof “firsts” - the first Functional Sizing methodto:– be designed by an international group of

experts on a sound theoretical basis– draw on the practical experience of all the

main existing FP methods– be designed to conform to ISO 14143 Part 1– be designed to work across MIS and real-

time domains, for software in anyarchitectural software layer


The COSMIC model showing the interactionThe COSMIC model showing the interactionof the Software Item being measured withof the Software Item being measured with

the ‘Boundary’ of its Layerthe ‘Boundary’ of its Layer

The ‘User’ (Person or any ‘thing’ including other Software)

Data Read Data Write

The Software Item = process

Data Manipulation

The ‘Boundary’ of theSoftware Item

Data ExitData Entry

Sub-Process Types -COSMIC countsthese to measure thefunctional size

Data Entries and Exits move data across the User/Software BoundaryData Reads and Writes store and retrieve persistent data within the

software item, as far as its ‘User’ is concernedVersion 1.5 4Q99 © COSMIC 199

Summary: the COSMIC FFP Meta-modelSummary: the COSMIC FFP Meta-model

SoftwareEnvironmentLayer

Transaction-Type(=Functional

Process)

Data Movement-Type( = FunctionalSub-Process)

User(-Type)

Event(-Type)

SoftwareItem

OutputEntry(-Type)

Write(-Type)

Exit(-Type)

Read(-Type)

Input

DataAttribute

(Type)

Object( = Entity-

Type )

DataGroup(Type)

(is one of)

(Moves)

Degree ofRelationship:

One-to-many

One-to-one

COSMIC-FFPBFC-Type

Triggers

Version 1.5 4Q99 © COSMIC 199

To download Counting Manualgo to Cosmic Web site :

www.cosmicon.com

25

© Total Metrics


• To introduce the COSMIC project,its aims and structure

• To describe the current status ofthe project and Interim findings

• To invite interested organisationsto participate in the next stage offield-testing of the COSMICprinciples


26

© Total Metrics

The Field Trial aim: toThe Field Trial aim: toadvance the Method’s statusadvance the Method’s statusfrom ‘proposal’ to ‘proven’from ‘proposal’ to ‘proven’

• to test for a common, repeatable interpretation ofthe COSMIC Version 2 Measurement Manualunder widely-varying conditions (organisations,domains, development methods, etc).

• to establish the detailed procedures, wherenecessary to ensure repeatable interpretation

• to test that the measures properly representfunctionality and/or correlate with developmenteffort

• to enable a full transfer of technology to the trial‘Partners’


27

© Total Metrics

Specific and subsidiarySpecific and subsidiaryaimsaims

• to obtain data from up to 20 Partners,and/or for 50 - 100 software developmentprojects

• to complete the trials by mid-2000 and topublish the ‘field trials proven’ version ofCOSMIC FFP, incorporating anyrefinements resulting from the trials

• to establish initial benchmarks for themethod

• to size the same software by conventionalFP methods, to examine convertibility

• to establish approximation rules for thefull COSMIC FFP method, for use in sizingsoftware early in its life


28

© Total Metrics

The Field Trials processThe Field Trials process

Initial Planning

•Gain commitment•Select projects

Data Collection

•Mainly a Partner task•COSMIC Team support

Preparation

•Training•Repeatability Exercise

Central Analysis (UQAM)

•Method refinements•Calibration

•Convertibility•Benchmarks

IndividualPerformance

Reports

Local/RegionalFeedback


29

© Total Metrics

COSMIC FFP V2 Field TrialsCOSMIC FFP V2 Field TrialsOrganisations ParticipatingOrganisations Participating

(as at April 2000)(as at April 2000)

Started via Graduate students• Hydro Quebec (Canada, power utility, process control)• Büren & Partner Software Design (German telecoms

software)

Started or starting soon• European Aerospace Co. and maybe a Nth American

(avionics software)• UK Bank (MIS systems)• Two European telecommunications manufacturers• European Telecommunications Operator• Canadian software house• Two Australian organisations(?)


30

© Total Metrics

The COSMIC FFP methodThe COSMIC FFP methodseems to be equally applicableseems to be equally applicableto real-time software as to MISto real-time software as to MIS

softwaresoftware• IFPUG: classifying Elementary Processes as

External Inputs, Outputs or Inquiries is OK inthe MIS world, but often difficult for real-timesoftware; the weights do not seem appropriate

• COSMIC FFP: the Measurement Model ofFunctional Processes decomposed into DataMovements (Sub-processes), seems equallyeasy to apply to MIS and real-time software

• Productivity measures indicate higher unitcosts for real-time and telecommunicationssoftware than for MIS


Bühren & Partner has developedBühren & Partner has developeda simple estimating formulaa simple estimating formula

based on COSMIC FFPbased on COSMIC FFPCOSMIC FFP productivity measures fit a COCOMO-like formula:

CFFP Productivity = a + b * (Size) n

Estimation Method

Absolute deviation range

Mean deviation

Standard deviation

Traditional /Expert

-50% to +50%

-14%

34%

CFFP Prod.Function

-30% to +33%

-2.3%

17%

ArtemisKnowledgePlan

-20% to +46%

+1.2%

26%

Notes:

Some project data used to calibrate productivity formula, which was then used to predict effort for other projects

KnowlegePlan has not been set up for CFFP measurements and its performance can be improved by bettercalibration

Buhren will continue to use CFFP for productivity measurement and to improve its estimating (DSMA Fall 99Conference) Version 2 April © COSMIC 2000

32

© Total Metrics

Bootsma’s IFPUG + FFPBootsma’s IFPUG + FFPestimating model’sestimating model’s

accuracy is impressiveaccuracy is impressive

Estimate Error Distribution

0

2

4

6

8

10

<-44%<-36% <-20% -12%to-4%

Estimate Error percent

Fre

qu

ency

IFPUG + FFP

Standard deviations of Estimate Errors:

Only IFPUG = Std Dev of 22%

IFPUG + FFP = Std Dev of 7%

-4%to+4%

+4%to+12%

>+28%>+20%

IFPUG


33

© Total Metrics

Proposals have been made forProposals have been made foran approximate early CFFPan approximate early CFFP

sizing methodsizing methodMacro Process(= System?)

General Process(= Sub-system?)

FunctionalProcess

For example the set of Create,Read, Update and Delete (‘CRUD’)

functional processesfor the life-cycle of a data entity

• Early life-cycle estimating is of great economicimportance

• Research has started on how to defineunambiguously and to size higher-level groups ofprocesses for early estimatingR Meli, A Abran, Vinh Ho, Serge Oligny - in preparation


34

© Total Metrics

The initial, general feedback isThe initial, general feedback isvery positivevery positive

• ‘Easy to measure without being a measurementexpert’

• ‘The functional sizes measured with COSMICFFP for each of three parts (of a system) alignedwith the perceived distribution of functional size’

(ESI Software Inc., Canada)• ‘Project Teams were able to grasp the elements

of the method easily and were enthusiasticabout the method’

• ‘Documentation and effort needed is similar tothat for applying the IFPUG method, thoughthere is an extra step to identify layers’

(UK Bank)• ‘Results (are) so promising, I am sure we will

continue the measurements…’(European participant)


35

© Total Metrics

Research activities aroundResearch activities aroundCOSMIC FFPCOSMIC FFP

• Inter-measurer consistency study (Patrice Nolin, UQAMwith Hydro Quebec);

• Conversion from FFP V1, MkII and IFPUG (VinhHo,UQAM)

• Early COSMIC-FFP (Chapter 7) - UQAM & R. Meli (Italy)• Correlation of expert view of functionality with COSMIC

FFP size, using AHP (Gerhard Wittig & EberhardRudolph, Australia)

• Procedure for UML-based specifications (Valerie Bevo,UQAM)

• Automatic measurement from source code (Vinh Ho,UQAM)

• Size contribution of Technical and Quality requirements(Chris Lokan, Australian Defence Academy & UQAM)

• Other aspects of size - algorithmic complexity Kececi(USNRC), Bootsma, (Nortel) planning to study

• Supporting requirements identification with CBRapproach (Jean-Marc Desharnais, UQAM)


36

© Total Metrics

COSMIC FFP Tools, ISBSGCOSMIC FFP Tools, ISBSGbenchmarksbenchmarks

• Hierarchy Master - FFP v. 1 fullysupported, V. 2 in development(Jin Ng, Australia)

• Sphera (Italy) - measurementsupport and estimating tool for V.2 in development (Roberto Meli,Summer 2000)

• Commitment to deliver Field Trialresults to ISBSG (and to trialparticipants)


37

© Total Metrics

There is strong internationalThere is strong internationalinterestinterest

• The COSMIC Measurement Manual has beentranslated into French, Italian, Japanese,Spanish. German to come

• The Measurement Manual has been down-loaded from over 30 countries

• We continue to present talks about COSMICFFP at international conferences, e.g– ESCOM April– ESEPG Amsterdam, June– IWSS 2000, Berlin, October– FESMA Madrid, October– Australia, Japan and North America

• The French Association pour l’Etude deMétriques en Informatique is setting up a StudyGroup on COSMIC FFP


38

© Total Metrics

And planning furtherAnd planning furtherahead…..ahead…..

A proposal has been submitted bythe Canadian National Body toISO/IEC/JTC1 SC7 (SoftwareEngineering) for a New Work Item tostart putting the COSMIC FFPmethod through the ISOstandardisation process workingwithin the framework of ISO 14143


39

© Total Metrics

Conclusion - we’reConclusion - we’remaking great progress!making great progress!

• The acceptance from those who have tried themethod seems good in both MIS and real-timeenvironments

• All the questions that have been raised havebeen solved , however we need– to do more work on defining layers– decide the final weightings

• Most organisations are taking longer to getstarted and to collect data than we had hoped,but we are getting there

• Our project has stimulated a great deal ofresearch

The COSMIC Core team would like to thank the trialThe COSMIC Core team would like to thank the trialparticipants, the researchers, especially in UQAM, andparticipants, the researchers, especially in UQAM, andothers who have helped for their support and interestothers who have helped for their support and interest..


40

© Total Metrics


• To introduce the COSMIC project,its aims and structure

• To describe the current status ofthe project and Interim findings

• To invite interested organisationsto participate in the next stage offield-testing of the COSMICprinciples


41

© Total Metrics

We need your help!We need your help!

The COSMIC Team is seeking JAPANESEsoftware producers or users in the MIS andreal-time domains who:– can appoint a software measurement

expert to review the applicability of theCOSMIC approach in their organisation

– can provide example softwarerequirements and development effort datato help test and calibrate the COSMICmethods with support from a COSMICTeam member

– can contribute financially to the projectcosts


42

© Total Metrics

Participants in the COSMICParticipants in the COSMICfield-testing should gainfield-testing should gainsignificant benefitssignificant benefits

• Early access to the new ideas and theopportunity to influence them

• Measurement of own pilot projectswith the new method:

– measurement data

– confidential report of corporatemeasurements

– mentoring during pilot projects &results analysis

• Pre-release technology transfer duringpilot projects


43

© Total Metrics

Organisations wishing toOrganisations wishing toparticipate in the field-testingparticipate in the field-testing

phase should contact anyphase should contact anyCOSMIC Core Team memberCOSMIC Core Team member

Version 1.5 4Q99 © COSMIC 199

Principle contact addresses:

Web-site: www.cosmicon.com

Alain Abran: [email protected]

Charles Symons: [email protected]

Pam Morris: [email protected]

Further details: www.lrgl.uqam.ca


44

© Total Metrics

Thank YouThank Youforfor

ListeningListening

For more information please email :[email protected]


Documents

Software Measurement Now and the Future · ( International Function Point User Group) ... ( COmmon Software Measurement International Consortium) – research, development and trials