Software Metrics and Measurements

University of Southern California

Center for Systems and Software Engineering

Supannika KoolmanojwongCS510

Outline

• General Concepts about Metrics• Example of Metrics• Agile Metrics• Metrics from Empirical Data

Measurements in daily life

Why do we measure?

Objectives of software measurement

• “You can not control what you cannot measure.” – Tom DeMarco

• “Not everything that counts can be counted. Not everything that is counted counts.” – Albert Einstein

Software Metrics

• Numerical data related to software development

• Strongly support software project management activities

• Can be directly observable quantities or can be derived from one

A simplified measurement information model

7Ref: Ebert and Dumke 2007

Decisions / Actions Measurements

Process Work Products

Information products

Information needs

Attributes

Results

Information Needs,

Objectives, Control

How the software measurements are used?

• Understand and communicate• Specify and achieve objectives• Identify and resolve problems• Decide and Improve

Measurement Standard

ISO/IEC 12207 Software Life Cycle Processes

ISO/IEC 15288 System Life Cycle processes

SWEBOKSoftware Engineering Body of Knowledge

PMBOKProject Management Body of Knowledge

CMMICapability Maturity Model Integration

ISO 15504Software Process Capability Determination

ISO 9001Quality Management System

ISO/IEC 9126Software Product QualityTL 9000, AS 9100, etc.Objectives adaptations

How to do How to do better

ISO/IEC 15939:2002Software Measurement Process

How to measure what you are doing

Ground rules for a Metrics• Metrics must be

– Understandable to be useful– Economical– Field tested– Highly leveraged– Timely– Must give proper incentives for process improvement– Evenly spaced throughout all phases of development– Useful at multiple levels

10http://www.stsc.hill.af.mil/resources/tech_docs/gsam3/chap13.pdf

Measurements for Senior Management

• Easy and reliable visibility of business performance

• Forecasts and indicators where action is needed

• Drill-down into underlying information and commitments

• Flexible resource refocus

Measurements for Project Management

• Immediate project reviews• Status and forecasts for quality, schedule,

and budget• Follow-up action points• Report based on consistent raw data

Project management supporting metrics1. Planning - Metrics serve as a basis of cost

estimating, training planning, resource planning, scheduling, and budgeting.

2. Organizing - Size and schedule metrics influence a project's organization.

3. Controlling - Metrics are used to status and track software development activities for compliance to plans.

4. Improving - Metrics are used as a tool for process improvement and to identify where improvement efforts should be concentrated and measure the effects of process improvement efforts.

Measurements for Engineers

• Immediate access to team planning and progress

• Get visibility into own performance and how it can be improved

• Indicators that show weak spots in deliverables

• Focus energy on software development

The E4-Measurement Process

Objectives, needs

Business Process

Environment, resources

1. Establish 2. Extract 3. Evaluate 4. Execute

Decisions, re-direction,

updated plans

Ref: Ebert and Dumke 2007

Aggregation of informationEnterprise

Division

Product Line/ Department

Projects

Cycle time, quality, cost, productivity, customer

satisfaction, resources, skills

Sales, cost reduction, innovative products, level of

customization

Cost reduction, Sakes, margins, Customer Service

Cash flow, Shareholder value, Operations cost

SMART goals

• Specific - precise• Measurable - tangible• Accountable – in line with individual responsibilities

• Realistic - achievable• Timely – suitable for the current needs

What do you want to measure?

• Processes– Software-related activities

• Products– Artifacts, deliverables, documents

• Resources– The items which are inputs to the process

Components of software measurements

Example of Metrics

• Progress / Effort / Cost Indicator• Earned value management• Requirements / Code Churn• Defect-related metrics• Test-related metrics

• How big is the healthcare.gov website? – http://www.informationisbeautiful.net/

visualizations/million-lines-of-code/

Size• Earth System Modeling Framework Project

22http://www.earthsystemmodeling.org/metrics/sloc.shtml

Progress Indicator

Effort Indicator

Cost Indicator

Earned value management• Planned Value (PV) or Budgeted Cost of Work Scheduled (BCWS)• Earned Value (EV) or Budgeted Cost of Work Performed (BCWP)

26http://en.wikipedia.org/wiki/Earned_value_management

Burndown Chart

27http://en.wikipedia.org/wiki/Burn_down_chart

Requirements Churn/ Requirements Creep/

Requirements Volatility• number of changes to system requirements

in each phase/week/increment

Code Churn• Software change history• Large / recent changes• Total added, modified and deleted LOC• Number of times that a binary was edited• Number of consecutive edits

Code Complexity• Gathered from code itself• Multiple complexity values• Cyclomatic complexity• Fan-In / Fan-Out of functions• Lines of Code• Weighted methods per class• Depth of Inheritance• Coupling between objects• Number of subclasses• Total global variables

Code coverage

• Degree to which the source code is tested• Statement coverage

– Has each node in the program been executed?• Branch coverage

– Has each control structure been evaluated both to true and false?

Code Coverage

32http://www.firstlinesoftware.com/metrics_group2.html

JUnit Code CoverageTool instruments byte code with extra code to measure

which statements are and are not reached.

Package Level Converage

A Code Coverage Report

Line Level Coverage

http://www.cafeaulait.org/slides/albany/codecoverage/Measuring_JUnit_Code_Coverage.html

Defect reporting metric

• Can be categorized by – Status

• Remaining / Resolved / Found– Defect Sources

• Requirements / Design / Development– Defect found

• Peer review / unit testing / sanity check– Time

• Defect arrival rate / Defect age

Defect Status

Defect Density

Test Pass Coverage

37http://www.jrothman.com/Papers/QW96.html

Defect Density

Defect Per LOC

Developer Code Review

After 60‒90 minutes, our ability to find defects drops off precipitously

http://answers.oreilly.com/topic/2265-best-practices-for-developer-code-review/

As the size of the code under review increases, our ability to find all the defects decreases. Don’t review more than 400 lines of code at a time.

http://answers.oreilly.com/topic/2265-best-practices-for-developer-code-review/

Top 6 Agile Metrics

42Ref: Measuring Agility, Peter Behrens

43Ref: Measuring Agility, Peter Behrens

Velocity = Work Completed per sprint

Measurements in organizational level

• Empirical analysis• Change from the top

46Richard W. Selby, Northrop Grumman Space Technology, ICSP '09Title: "Synthesis, Analysis, and Modeling of Large-Scale Mission-Critical Embedded Software Systems"

Measurements for progress vs predictionsProject Phase For Measurements For PredictionsProject Management

-Effort and Budget Tracking- Requirements Status-Task Status

-Top 10 risks- Cost to complete- Schedule evolution

Quality Management

-Code Stability-Open defects-Review status and follow up

-Residual defects-Reliability-Customer satisfaction

Requirements Management

-Analysis status -Specification progress

-Requirements volatility / completeness

Construction -Status of documents- Change requests-Review status

-Design progress of reqm- Cost to complete-Time to complete

Test Test progress (defects, coverage, efficiency, stability

- Residual defects- reliability

Transition, deployment

-Field performance (failure, corrections) - maintenance effort

-Reliability-Maintenance effort

49Ref: Ebert and Dumke, 2007

Recommended books

Software Measurement: Establish - Extract - Evaluate – Execute by Christof Ebert, Reiner Dumke (2010)

Practical Software Measurement: Objective Information for Decision Makers by John McGarry, David Card, Cheryl Jones and Beth Layman (Oct 27, 2001)

References• http://sunset.usc.edu/classes/cs577b_2001/metricsguide/metrics.html• Fenton NE, Software Metrics: A Rigorous Approach, Chapman and Hall,

1991.• http://www.stsc.hill.af.mil/resources/tech_docs/gsam3/chap13.pdf• Christof Ebert, Reiner Dumke, Software measurement: establish, extract,

evaluate, execute, Springer 2007• http://se.inf.ethz.ch/old/teaching/2010-S/0276/slides/kissling.pdf• Richard W. Selby, Northrop Grumman Space Technology, ICSP '09

Title: "Synthesis, Analysis, and Modeling of Large-Scale Mission-Critical Embedded Software Systems“

• Measuring Agility, Peter Behrens• [Nikora 91] Nikora, Allen P. Error Discovery Rate by Severity Category and

Time to Repair Software Failures for Three JPL Flight Projects. Software Product Assurance Section, Jet Propulsion Laboratory, 4800 Oak Grove Drive, Pasadena, CA 91109-8099, November 5, 1991.

Software Metrics and Measurements

Documents

Software Metrics

Software Metrics - IDATDDC88/theory/12Metrics.pdf · Software metrics • Usage-based metrics • Verification & Validation metrics • Volume metrics • Structural metrics • Effort

Measurements and metrics 10 11

Software Metrics Cmpe 550 Fall 2005. Software Metrics

Understanding Website Complexity: Measurements, Metrics, and

Software Metrics - Software Engineering

Software Metrics and Measurements

CHAPTER 10 – Software Metrics - UAntwerpenlore.ua.ac.be/Teaching/SE3BAC/slidesPDF2010/10Metrics.pdf · Software Metrics CHAPTER 10 – Software Metrics Introduction • When,

Measurements and Metrics Notes

Smart measurements and analysis for software quality ...Smart measurements and analysis for software quality enhancement 3 - the development of techniques and tools, Metrics Suggester

7. Metrics in Reengineering Context Outline Introduction Metrics and Measurements Some metrics Conclusion

Security/Information Assurance Measurements and Metrics IA Measures-Sept14.pdf · Security/Information Assurance Measurements and Metrics ... –Risk management standards –Metrics

Software Metrics

Software Quality Metrics Overview - Cal Polyusers.csc.calpoly.edu/~djanzen/setopics/metrics/SoftwareMetrics.pdf · Software quality metrics are a subset of software metrics that focus

Software measurement and software metrics in software qualitypooh.poly.asu.edu/Ser515/.../docs/MeasurementAndMetricsSoftwareQ… · Software measurement and software metrics in

Film metrics and measurements - mycourses.aalto.fi

Understanding Broadband Traffic - Metrics, Measurements…cfp.mit.edu/.../BAUER-Slides.CFP.Munich.2010.pdf · Understanding Broadband Traffic - Metrics, Measurements, and Policy Steven

Metrics & Measurements - STeP-IN Forum

Packet Network Timing Measurements, Metrics, and Analysis · Packet Network Timing Measurements, Metrics, and Analysis ... Packet selection processes and methods ... Network Measurements

Relative frequency measurements: Metrics for sample