Testing Leadership: From 2nd Class Citizens to Best of Breed

IBM Quality thru Software Engineering

29 August 2006 © 2006 IBM Corporation

Bill WoodworthDirector, IBM Quality thru Software Engineering

Testing Leadership:From 2nd Class Citizens

to Best of Breed

Keynote at TAIC PARTConference

© 2006 IBM Corporation2


Testing Leadership in IBM – TAIC PART Conference – 29 August 2006

Agenda1. Testing challenges to overcome in 1998

and pitfalls to watch out for…

2. IBM’s Response to these challenges

3. IBM wide Teaming of S/W Test Professionals

4. Results of Testers’ Leadership and Community

5. Keys to Making It All Happen…

Testing Progress in IBM




Testing Challenges – in 1998

Low Recognition of the Value2nd Class Citizens

Isolated Islands of TestersNot Knowing Best Practices & ToolsEach Doing Their Own Thing; Tool Proliferation

Unclear customer usage/expectations of new functionsNot clear how to really verify the new functionReal customer scenarios not tested

Lack of disciplined process between Development & TestEntry / exit criteria weak or not applied with disciplineLack of early involvement by testers; Late deliveries from dev

Test technology had lagged behind Development

$?

Industry & academiaLack of automation




Testing Pitfalls

1.Poor test planningBlown schedule Easiest items handled first

& High risk items later

3.Weak enforcement of entry & exit criteriaFalse starts, wasted timePoor quality code and

documentation on test entry

2.Poor test designTests redundant & inefficientProduct exercised - not verifiedWeak exception/boundary testsDifferent sequences not tested;

Error paths not well tested

4.System / Integration TestsOverlap between some testsHoles / gaps between others




IBM’s RESPONSE =

CREATED A COMPANY WIDE COMMUNITY OF TESTERS

VisionTeaming to Become BEST OF BREED

Mission Delivering VALUE to Each Business Unit




Key FOCUS AREAS

1. COMMUNITY Development

2. PEOPLE: Critical Skill / Competency

3. Selecting / Using PRACTICES & TOOLS

4. PROCESS and METRICS

5. Driving for IMPROVEMENTS




1. COMMUNITY DEVELOPMENT

Executives Raising Expectations – Key

Team of Impassioned LeadersTechnical and Management

First Principle = “IBM T.E.S.T.” Integrating Better Methods Together

Enhancing Software Testing

Do It Once

Deploy to Many

Many

Many

Do it Once….. Deploy it to ALL !




Highly Successful Software Communities•Drive and Provide the Means for:

Community Community

Developing Resourcesneeds basedcommittees +

workgroupsbest technology /

source

Driving Improvementsself assessmentusing community

resourcesremembering lessons

learned

Developing Peoplecareer & skill focus

training & recognition

ChangingDeploying

Gro

win

g

DeliveringValue

Sharingnurturing

communicationswebsite, T.R.s, DBs

newsletters; meetings

Networkingin-person meetings

building relationships

site visits; contacts

Leading & Decidingimpassioned unit

leaders‘change leaders’

TeamingInfo

rming

Gu

idin

g

OperationalInfrastructure




2. PEOPLE

Establish Testing as a Recognized Critical Skill and Competency

Ensure Testers Have or Obtain the Required Skills

Actions New recognition opportunities Tester career roadmaps & guidance Test education & skills development

via classes, ITU, conferences, etc.

Management / executive emphasis and feedback




TESTING: Professional Excellence with UNLIMITED Future

Skills Developed *Programming*Problem solving *Knowledge - product - system *Leadership *Creativity *Multi-languages

TESTERS:TESTERS:Customer Expectations

Marketplace needs..

System view: The Big PictureHow the products interact &

fit together; Breadth & Depth

Business functionality How used &

What it does & does not do

LEARNLEARN

Many Roles *Software Engineer *Solution architect *Sol’n consultant *Customer advocate *Systems engineer *Automation expert*Jack of all trades

A Multi-Technology

A Multi-TechnologyCareerCareer

A Multi-Industry

A Multi-Industry

ProfessionProfession




3. SELECTING Practices & Tools

Architecture Based on Eclipse + Rational

Best Technologies Labs: internal tools Research: teaming on next generation Using / enhancing IBM Rational products

Overall StrategyBecome Intensely Automated

Deployment = Challenging & Critical Practices based with supporting tools Creating resources + info once ‘available to all’ Saturating community with ‘right’ information Facilitating ‘how to’, successes, free consultants Marketing it….. Energizing local teams




4. Test Process Background

Test process = key element of product development process

As a company, we develop a lot of code: Some we delivery to our customers = ‘software’ Some we use to verify our software = ‘testware’

We are re-engineering the processes and practices applied by our testers and developers via IRUP IBM’s (customization of ) Rational Unified Process Selecting best practices from RUP, Test Community,

Development Community and IBM SMEs




Test Project Manager

Test Architect

Test Customer

Advocate

Test Designer

Test Implementer

Roles = Skills & Responsibilities

Test Discipline Workflow

4. Test Process – enhancing via IRUP




4. Closed Loop Process

Key = Learning & Remembering From Mistakes Disciplined & focused causal analysis . Orthogonal Defect Classification (ODC) .

Improving

EmphasisPlanningDesign / DevelopmentExecutionAutomation

Reducing escapes

TEST

Learning/Changing

Experiences fewer problems / defectsImproved solution availability and consumability

Learning/Changing

CUSTOMERS

Learning / Changing a Must

Customer

FocusReducing escapes

ProcessEase of UseAvailabilityReliability

Development




4. PROCESS - Metrics

● Critical RoleCritical Role in Improving S/W ProcessesFocus on Quality Process & Cost of Quality

Type of Key MetricsDelivered Quality MetricsIn Process Metrics

How Used…..Business Decision Making ProcessDriving Improvements

Used to demonstrate progress and value Causal Analysis (lessons learned & remembered) Process Self Assessments (by testers and now developers) Scorecards / Dashboards (lab or business area basis)




5. DRIVING for Improvements

Assessing where you are & where to improve… via annual Test Self Assessment

by each business unit

Ensuring process discipline applied

Using metrics to demonstrate Value

Gaining Executive Buy-in and Support

Motivating Test Leaders to be ‘Change Leaders’‘Change Leaders’ ‘System Test’ as ‘First Customer’ Drive across Test AND Development




5. Two Keys for Unit Self Improvement

1. ITCL Unit Leaders Aggressive ‘Change Leaders’

2. Meaningful and disciplined use of the IBM Test Self Assessment to affect the most beneficial improvements – business wise

SoftwareDevelopment

Process

TestSelf-Assessment

Improvement Actions

IRUP

IBM Best Practices

Identify Strengths and WeaknessesDevelop and Implement

Improvement Plan

Review with VP;Track and

Assess Value

Begin Improvement

Process




Agenda1. Testing challenges to overcome in 1998

and pitfalls to watch out for…

2. IBM’s Response to these challenges

3. IBM wide Teaming of S/W Test Professionals

4. Results of Testers’ Leadership and Community

5. Keys to Making It All Happen…

Testing Progress in IBM




THE RESULTS

PeoplePeopleUNLIMITED CAREERSUNLIMITED CAREERS

BOTH Higher Quality& Available Sooner

Product SolutionsProduct Solutions

Major Test Effectiveness& Efficiency Gains

BusinessBusiness

LeadershipLeadershipLeading Unit’s Improvement Drive…Leading Unit’s Improvement Drive…




THE RESULTS

LeadershipLeadership

•Within Each Product Team, Testers: teamed with developers involved from the start part of architecture / design efforts

•As ‘Change Leaders’, Testers: led the charge for improvements from requirements thru delivery

•Test = Foundation for total SW Eng’g focus




Test Foundation for SW Engineering Focus

2003Software Developers Community Leadership

1998 IBM Test Community Leadership

1998SoftwareQualityChampions

1998: Created S/W Testing Process & Quality FocusesCharged by Corporate Technical

Committee (CEO, Senior Execs) Formed IBM S/W Testers

Community and Software Quality Champions

2003: Added Development Process Focus:Because of and based on

success of Test CommunityCommissioned by IBM S/W

and Corp Tech Senior ExecsFormed IBM S/W Developers

Community

2005: Unified 3 into 1 QSE




Quality thru Software Engineering Community

QSE Community Vision

All Units are Better Achieving their Business Goals by Applying Better Quality Software Engineering !

(customer sat, speed to market, time to value, profit contribution, etc.)

QSE Community MissionTeaming to Improve Units’ Software Operations

and our Clients’ Satisfaction !




QSE Transforming Our S/W Engineering

Team Building & Networking

Developing QSE Resources

Deploying QSE Resources

Driving QSE Improvements…….

Re-Engineering via IRUP

_1998_ _1999_ _2000_ _2001_ 2002-04 _2005_Type text

Type text

Type text

Typ

e t

ext

_2006_

Yielded 100s of Improvement Actions and $10,000,000s of Benefits Each Year




KEYS TO SUCCESS THRU TEAMING

●NetworkingNetworking●TrustingTrusting●TeamingTeaming

●DeliveringDelivering Resources Resources

●DevelopingDeveloping Resources Resources

●DrivingDriving Improvements Improvements

●DeployingDeploying Practices Practices

VALUE

TO MAKE REALTO MAKE REALPROGRESS TAKES:PROGRESS TAKES:

Executive Support

TimeImpassioned

drivePersistence

Documents

Testing Leadership: From 2nd Class Citizens to Best of Breed