Balancing Your Test Effort

Planning test with Google’s approach

Nikita Knysh

nik@scanjour.dk

Ciklum, August 17, 2011

The Speaker

• Nikita Knysh, 30 y.o., ~11 years in IT

• Background: IT education, webdev, support+lead, PM, TW, BA+lead, ISTQB FL

• Now: 4 years with Scanjour, Test TL

Agenda

• The Challenge• The Directions• The Method• The Tools• Probs & Cures

The Challenge

• What to test if not enough time?• What to test if have buffer?• Much test, few bugs• No overview of test needs• Not enough sync between roles

The Directions

• Discuss• Summarize• Prioritize

… team-wide & easily

So what’s the right approach then??

The Method: Origin

ACC (Attribute-Component-Capability)

James A. Whittaker,

Test Director at Google,

2010

Microsoft and Google use this.

Can we?

The Method: Concept

• List product’s selling points (Attributes)• Break down the product into tech

Components. • Break down the product based on

WHAT it does (Capabilities).

We get a model that reflects all

the vital views on the product!

The Method: ACC Modeling

• ACC list

• Time to give it some numbers!

The Method: Giving it Numbers

• Complexity

increases risk of human mistakes during code development and maintenance and therefore risk of introducing bugs

• Use full scale (1 to 5)• Track averages

The Method: Giving it Numbers

• Complexity factors

code logicbusiness logic

how much UI?

how much error handling?

how easy to debug?quality of requirements

tech expertise

unit test coverage

auto test coverage

code ownership

architecture

The Method: Giving it Numbers

• Frequency of Use

how often the capability is called by user or automatically and therefore how often failures caused by defects in code will most likely occur

The Method: Giving it Numbers

• User Impact

damage dealt to user and / or system intent should the capability fail completely or severely

The Method: Outcome

• Testing Needs

= Complexity * Frequency * Impact

Now we know where the risk is.

Now we know where we need more test.

and…

Our knowledge is based on cumulated vision of the whole team!

…and it is extremely easy to overview!

The Method: Outcome

• ACC list, now with numbers

The Method: Outcome

• Matrix view of capability count

The Method: Outcome

• Matrix view of testing needs

The Tools: How We Do It

• ACC items are TFS work items• ACC linked to TCs and bugs for metrics• Excel book for each model• Two-way sync between TFS and Excel• Instant update: DWH cube is avoided• Pivot tables and charts

make the beauty

The Tools: How It Looks

• Model overview in Excel

That easy?

Really?

Probs & Cures

• “No time for modeling!”. Include into DoD.

• “It shows what we knew!”

Be strong! Don’t manipulate!

+True for any document.

More Probs & Cures

• Can’t test individual capabilities. Reconsider product breakdown.

• Hard to bind test cases and bugs to capabilities. Reconsider model’s detail level. Think starting from ACC, not test cases.

Thank you!

• Questions & Answers