IEEE Keynote 2004 09

8/2/2019 IEEE Keynote 2004 09

1/25

"Using Metrics to Drive QualityImprovements and Value --

Company Wide"

Bill WoodworthDirector, IBM Software Test &

Development ExcellenceIBM Corporation

[email protected]

Presented at IEEE ICSM Conference

13 September 2004

8/2/2019 IEEE Keynote 2004 09

2/25

8/2/2019 IEEE Keynote 2004 09

3/25

IBM S/W Test & Development Excellence

Using Metrics to Drive Quality Sept 2004 -- IEEE Copyright 2004 IBM Corporation3

IBM is a Software Company

In fact, IBM is a HUGE SOFTWARE Company

a) Our products span the entire range: firmware / microcode,operating systems, middleware, systems management,applications, imbedded, etc.

b) Software revenue = $14.3 B in 2003; 1/3 of total company profits

c) World's largest infrastructure (middleware) software provider

d) Second largest software business in the worlde) #1 or 2 in market share in all our software markets (per IT analysts)

f) High double digit thousands of software engineers

g) Thousands of software products

And QUALITY Matters Big Time to Our Customers IBM's systems are generally NOT desktops where you can afford

to crash, burn and re-boot BUT are business-critical systemsthat MUST NEVER crash

Holistic Quality is key to our Success as a Software provider

8/2/2019 IEEE Keynote 2004 09

4/25



Our Focus on Holistic Quality

Quality is key to IBMs success as a software provider

Total Customer Satisfaction in our products and solutionsContinual improvement = each release better than its predecessor

We apply a quality management system balancing:Quality + Function + Speed to Market + Cost

where each aspect plays an important role

in delivering value to our customers

Function

Quality

Cost

Speed To Market

Software Quality focus must be Holistic:More than reducing defects

Ensuring Correctness

Increasing Availability, Reliability, Integrity, Installability, Testability Improving Problem Determination, Usability, Interoperability,

Portability, Efficiency, ....

Reducing and Preventing Problems

Meeting customer expectations

Disciplined risk management

8/2/2019 IEEE Keynote 2004 09

5/25



Our Quality System Before 1998

Each unit working to improve Quality of its products

Each release expected to be better than its predecessor

BUT limited company wide sharing on:Best processes and practices Lessons learned

Networking and helping each other

Since then, related complexities began really increasing:Economic need for more effective & efficient operationsQuality expectations of customers in the industryComplexity of technologies and combinations in solutions

New initiatives in OnDemand and Autonomic ComputingDevelopment and Test process pressures from all the above

Need and Opportunity for Software Quality EngineeringImprovements in ALL Units Company Wide SQE Focus

8/2/2019 IEEE Keynote 2004 09

6/25



Using Metrics to Drive Quality Improvements & Value

Agenda:Background


Driving & Leading Company Wide SQE Initiatives

Why = ObjectivesWhat = Communities of Testers and of Developers

How = Focuses and Success Factors

Using Metrics to Drive Improvements ValueCritical Role of and Types of Metrics

How Used

More Partnership across the Industry

8/2/2019 IEEE Keynote 2004 09

7/25




Each Community has Same Basic Vision & Purpose:Vision = Teaming to Become BEST OF BREED

Purpose = Delivering VALUE to Each Business Unit

In 2003: Added Development Process Focus

Created a company wideS/W Developers Community

Modeled on success & operations of ITCL

In 1998: Created Testing Process Focus

Created a company wideIBM Testers Community

Spans all product labs, global services, research

8/2/2019 IEEE Keynote 2004 09

8/25



Company Wide SQE Initiatives Focus

What we did & are doing = Key Focus Areas People and Community

Processes, Practices and Tools

Metrics and Driving for Improvements

How we were successful = Keys to Success

Recognized need to improve Recognized need to help each other

Unit Leaders = active and empowered

Value demonstrated to all players

Walk before you run mentality

8/2/2019 IEEE Keynote 2004 09

9/25



Highly Successful SQE CommunitiesDrive and Provide the Means for:

CommunityCommunity

Developing Resourcesneeds basedworkgroups

best technology/source

Driving Improvements

self assessmentusing best resources

remembering lessons learned

Developing Peoplecareer & skill focustraining & recognition

DeliveringValue

Sharingnurturing communications

website, T.R.s, DBsnewsletters; meetings

Networkingin-person meetingsbuilding relationships

contactsLeading & Decidingimpassioned unit leaders

change agents

OperationalInfrastructure

8/2/2019 IEEE Keynote 2004 09

10/25



Rational UnifiedProcess

IBM RationalProducts

UNIT

Current positionIssues & needs

UNIT

MeasurablyImproved

ITCL & SDCL

Resources

Test & DevTest & Dev

SelfSelf

Assess'tAssess't

Units Becoming Best of Breed

2 Keys within Each Unit Enabling Unit Self-Improvement1. ITCL & SDCL Unit Leaders Aggressive Change Agents

2. Meaningful, disciplined and supported application of the IBMTest and Development Self Assessmentsto affect

the most needed improvements

8/2/2019 IEEE Keynote 2004 09

11/25



TESTING: From Ideas to Implementation.

Team Building & Networking

DevelopingITCL Resources

DevelopingIBM Test Solutions..

DeployingITCL Resources

DeployingIBM Test Solutions

Driving TestImprovements.

Shifting toRational

_1998_ _1999_ _2000_ _2001_ _2002_ _2003_ _2004_

Yielded 100s of Improvement Actionsand $10,000,000s of Benefits Each Year

8/2/2019 IEEE Keynote 2004 09

12/25




Agenda:Background






How Used


8/2/2019 IEEE Keynote 2004 09

13/25



PROCESS METRICS

Critical Role in Improving S/W Development

Quality Process Focus & Cost of Quality

Type of Key MetricsDelivered Quality Metrics

In Process Metrics

How Used..Business Decision Making Process


Causal Analysis Process Self Assessments

Scorecards / Dashboards

Used to Demonstrate Progress and Value

Gaining Executive Buy In

8/2/2019 IEEE Keynote 2004 09

14/25



Delivered Quality Metrics

Number of problems whichcustomers report

Basic comparisons whichdemonstrates a given releaseis better than its predecessor

Trend of problems requiringintervention outside thenormal support process

Cost of handling customerreported problems/defects,sometimes normalized byrevenue or dev expense

Number of problems whichcustomers report which

result in code fixes

Number of system failures,usually normalized by usagemonths

Reported survey results

Timing = monthly, quarterly, annuallyOptional grouping by product maturity

Customer Satisfaction

Customer Reported Problems

Customer Reported Defects

Release-to-Release

Crit Sit Trend

Availability and OutagesWarranty / Service Costs

8/2/2019 IEEE Keynote 2004 09

15/25



In Process Metrics

Process to aid test teams inevaluating their units

strengths / weaknesses

improvement actionsProcess to aid developmentteams in evaluating theirunits strengths / weaknesses

improvement actionsAnalyzing defects / problems =

lessons learned & remembered

Means to understand how

much function or codehas been tested

Means to understand progress vs plan schedule (attempts & completions)

problems / defects (valid & total) outage durations defect arrival rates and density test effectiveness

Design / Code Complete

Functional Test

System Test

Coverage

Causal Analysis

Development Self-Assessment

Test Self-Assessment

Means to understand progress vs plan

schedule, size, functionality design / code peer reviews performed unit test completed

8/2/2019 IEEE Keynote 2004 09

16/25



PROCESS METRICS

Critical Role in Improving S/W Development

Quality Process Focus & Cost of Quality

Type of Key MetricsDelivered Quality Metrics

In Process Metrics

How Used..Business Decision Making Process


Causal Analysis Process Self Assessments

Scorecards / Dashboards

Used to Demonstrate Progress and Value

Gaining Executive Buy In

8/2/2019 IEEE Keynote 2004 09

17/25



Integrated Product Development (IPD) A management system designed to optimize the development and

the delivery of successful quality products and offerings.

Fact BasedData to ManageYour Business

Project Management Data

Segment Performance Data

Integrated Portfolio Management Team(IPMT)

Market Planning

Information

CustomerFeedback

CompetitorInformation

TechnologyTrends

CurrentProductPortfolio

Understandthe

Marketplace

PerformMarket

Segmentat'n

PerformPortfolioAnalysis

DevelopMarket

Strategy& Plan

Align &OptimizeBusiness

Plans

Manage Market Segment andAssess Performance

StructuredProduct

Development

Product ObjectivesBusiness Strategy

Customer Buying Behavior

R&D Roadmap

Candidate Projects

SatisfiedCustomers

$$$$

Life

CycleLaunchQualifyDevelPlanConcept

Project Development Teams

Resource Allocation

Project Management Data

Performance Data

8/2/2019 IEEE Keynote 2004 09

18/25



Integrated Product Development (IPD)

Concept

DCP

Plan

DCP

Availability

DCP

EOL

DCP

Strategy OfferingInvestment Execution

ConceptConcept PlanPlan DevelopDevelop QualifyQualify LaunchLaunch LifecycleLifecycle

IPMTand PDTContract

Event Driven Formal 'Decision Check Points' (DCPs)

B S

8/2/2019 IEEE Keynote 2004 09

19/25



Closed Loop Development Process

Key = Learning & Remembering From MistakesDisciplined & focused causal analysis .Orthogonal Defect Classification (ODC) .

Improving

Emphasis PlanningDesign /DevelopmentExecution

Reducingescapes

TEST

Learning/Changing

Experiences fewerproblems / defectsImproved solutionavailability

Learning/

Changing

CUSTOMERS

Learning a Must

Customer

FocusReducingescapes

Process

Ease of UseAvailabilityReliability

Development

IBM S/W T & D l E ll

8/2/2019 IEEE Keynote 2004 09

20/25



Used to Demonstrate Value & Progress

Metrics used to understand an aspect of the business

Not metrics for metrics sakeMeans to understand: WHERE are we?

Are we making PROGRESS?

Improvements in Process / PracticesIts all about trends

Progress in what.Customer satisfaction

Customer support

Reducing customer found defects and problems

Benefits of process / practice improvements made.

..

What is managements business / operational focus?Align metrics to develop applicable understanding

IBM S/W T t & D l t E ll

8/2/2019 IEEE Keynote 2004 09

21/25



Gaining Executive Buy-In to Drive Improvements

3 APPROACHES(in addition to base line management responsibility)1. Tops Down =

Corporate Guidance on Quality of Offerings

2. Peer Pressure ++ =Semi-annual Improving S/W Quality Executive Reviews

3. Partnering =Teaming Line Executives with Executives of

Software Communities

Function

Quality

Cost

Speed

IBM S/W T t & D l t E ll

8/2/2019 IEEE Keynote 2004 09

22/25



Teaming Line Executives with Community Executives

Line execs responsible for quality of their offerings

Holistic qualityplus operational effectiveness and efficiencies Must inspect what they expect

Active support for Process Self Assessments Set Expectations == Improving is not only OK, but is expected

Communities provide resources (single sourced & anticipatorily) Unit leaders have opportunity to demonstrate leadership

(i.e. look good) and drive improvements from business wise

SQE Community Execs with company wide leadership team Create IBM wide communication fabric

Develop needed resources onceSO ALL UNITS, who need them, can use them

TEAMING TO BECOME BEST OF BREED

IBM S/W Test & De elopment E cellence

8/2/2019 IEEE Keynote 2004 09

23/25




Agenda:Background






How Used



8/2/2019 IEEE Keynote 2004 09

24/25



More Partnership Across the Industry

In spite of industry wide efforts in using metrics to

drive quality, difficult problems still exist (e.g.):

More meaningful and easily collected customer impact metrics

Tools to more easily process multi-language legacy products

Automated test technologies (auto generation, etc.)

Approaches on how to turn academically sound metricsinto successful real world usage

The whole industry should work together to solve

these problems maybe you can help!

8/2/2019 IEEE Keynote 2004 09

25/25

Documents

IEEE Keynote 2004 09