Testing Does Not Equal Quality

Testing != QualityRethinking quality and how to achieve it in an agile world

Bob HartmanRichard Lawrence

“How does QA fit into all of this?”

are conflicting goals

Speed/value

quality

and

Or are they?

How can quality enable speed and value?

What is quality?

How do we ensure our software has it?

What is quality?

The goal of a software org:Produce valuable software now and in the future.

A feature can be modeled with a cumulative cash flow chart

Jan-09

Feb-09

Mar-09

Apr-09

May-09

Jun-09

Jul-09

Aug-09

Sep-09

Oct-09

Nov-09

Dec-09

-$300,000

-$200,000

-$100,000

$0

$100,000

$200,000

$300,000

$400,000

$500,000

$600,000

Or, we can remove the investment and just look at value flow

Jan-09

Feb-09

Mar-09

Apr-09

May-09

Jun-09

Jul-09 Aug-09

Sep-09

Oct-09

Nov-09

Dec-09

Jan-09

Feb-09

Mar-09

Apr-09

May-09

Jun-09

Jul-09 Aug-09

Sep-09

Oct-09

Nov-09

Dec-09

Value flow for a feature that never works right

Value flow for a feature that fails for some users some of the time

Jan-09

Feb-09

Mar-09

Apr-09

May-09

Jun-09

Jul-09 Aug-09

Sep-09

Oct-09

Nov-09

Dec-09

Value flow for a feature with downtimes

Jan-09

Feb-09

Mar-09

Apr-09

May-09

Jun-09

Jul-09 Aug-09

Sep-09

Oct-09

Nov-09

Dec-09

Value flow for a feature that doesn’t scale

Jan-09

Feb-09

Mar-09

Apr-09

May-09

Jun-09

Jul-09 Aug-09

Sep-09

Oct-09

Nov-09

Dec-09

Quality is the foundation of value

A team’s productivity can be measured with a velocity chart

Jan-09

Feb-09

Mar-09

Apr-09

May-09

Jun-09

Jul-09 Aug-09

Sep-09

Oct-09

Nov-09

Dec-09

Velocity of a team with a well-designed, high quality system

Jan-09

Feb-09

Mar-09

Apr-09

May-09

Jun-09

Jul-09 Aug-09

Sep-09

Oct-09

Nov-09

Dec-09

Velocity of a team with a poorly-designed, low quality system

Jan-09

Feb-09

Mar-09

Apr-09

May-09

Jun-09

Jul-09 Aug-09

Sep-09

Oct-09

Nov-09

Dec-09

Quality is the foundation of speed

Photo credit: woodsy on sxc.hu

Quality is what ensures: Value now and in the future

How do we ensure our software has it?

Testing at the end doesn’t workIteration Start Iteration End

The Plan

The Reality

Plan Dev Test

What’s really going on?

The Plan

The Reality

Dev Test

Dev Test

Dev expands to

fill the available

time

Dev to test is really a loop

Build quality in vs. inspecting for it at end

Inspect for quality Dev TestPlan

Build quality in

Manual testing doesn’t scale

Sprint 1 Sprint 2 Sprint 3 Sprint 4

New feature testing

Sprint 5

Testing that doesn’t happen,

but should

(or

happens

in

overtime)

Regression testing

Testing capacity

Automate and get regression tests for free

Sprint 1 Sprint 2 Sprint 3 Sprint 4

New feature testing

Sprint 5

Automated tests that are now

regression tests

Testing capacity

What is really being tested?

1 2 3 4 5 Total1 5

2 4

3 3

4 2

5 1

IterationsFe

atur

es

Which features need to work best?

Never Used45%

Rarely19%

Sometimes16%

Often13%

Always7%

Source: The Standish Group

Usage of features in a typical software product

But without regression it is different

1 2 3 4 5 Total1 1

2 1

3 1

4 1

5 1

IterationsFe

atur

es

Minimal regression is a little better

1 2 3 4 5 Total1 3

2 3

3 3

4 2

5 1

IterationsFe

atur

es

But what will happen where we don’t have regression tests running?

1 2 3 4 5 Total1 3

2 3

3 3

4 2

5 1

= Missed tests, which Murphy’s Law says will lead to defects being missed!

IterationsFe

atur

es

Build and deploy processes can kill speed

Define acceptance criteria and automate

tests

Develop Package Deploy to test

env

Test

DONEThe Plan

The Reality Loops, waits, errors…more time

To ensure quality in an agile world:

• Move testing to the front of the process• Collaborate• Automate tests (using agile-friendly tools)• Automate/accelerate build and deploy• Work on user stories one at a time, all the way

to DONE

Questions?

www.richardlawrence.inforichard@humanizingwork.com

www.agilebob.combob.hartman@agileforall.com