Agile Testing isn’t Risking IT!

Agile Testing isn’t Risking IT!

Wednesday 7 November 2012Eurostar 2012

2© All rights reserved

Agile Testing

isn’t Risking IT!

Introduction

What is it all about?

Which problems did I face?

How to do it right?

Recap-

Q & A

12345

6


Agile Testing

isn’t Risking IT!

Introduction


Which problems did

I face?

How to do it right?

Recap-

Q & A

12345

6

© All rights reserved

Agile in a nutshell

4

=

ProductBacklog

SprintBacklog

Sprint2-4 wk

Scrum24 u

Product Owner Team Scrum Master

DemoPlanning Meeting

Retrospective


“A process or method to analyse the risks to the product, with the intention to utilize the testing effort in the most valuable way.”

Product Risk Analysis in a nutshell

Classification: High

Classification: Medium

Classification: Low

Product RiskRequirement

“The likelihood that a product fails in relation to the conceivable damage when a product fails in production.”

5


Risk & Requirement Based Testing in a nutshell

Risk without requirement:• Add requirement (a fault found

early)• Remove Risk (no worries about

nothing)

Requirement without Risk• Add product risk (improves

coverage of test)• Remove Requirement (only

make what is needed)

Requirements

Product Risks


Agile Testing

isn’t Risking IT!

Introduction


Which problems did I face?

How to do it right?

Recap-

Q & A

12345

6


Risk analysis were not translated to the Backlog which makes it difficult to utilise a risk based approach.

Merging or splitting features invalidate the PRA.

SPECJ

SPECI

SPECH

SPECG

SPECF

SPECE

SPECD

SPECC

SPECB

SPECA

REQ02

REQ06

REQ04

REQ10

REQ09

REQ05

REQ01

REQ08

REQ07

REQ03

?

?

?

8


Risk analysis were not updated every iteration, which could mean that they will be out dated within a sprint.

PRA isn’t updated

REQREQREQREQREQREQREQREQREQREQ

PRPR


PRPRPRPR


PRPR

PR

PR

9


Risk analysis were only done from the requirements which meant that some product risks were missed.

PRA are done with tunnel vision

SpecifyTest Cases

DetermineRisk

OrderRequirements

AnalyseRequirements

Analyse Risk

Analyse Risk

OrderRequirements

AnalyseRequirements

SpecifyTest Cases

10


Risk analysis were done, but the outcome wasn’t taken into account in the planning meetings.

Risk class isn’t used in planning meeting

2 SP

4 SP

4 SP

2 SP

2 SP

6 SP

3 SP

2 SP

2 SP

8 SP

2 SP

3 SP

4 SP

6 SP

2 SP

Team Velocity=

± 14 Story Points

1

2

3

42 SP

4 SP

4 SP

2 SP

2 SP

6 SP

3 SP

2 SP

2 SP

8 SP

2 SP

3 SP

4 SP

6 SP

2 SP

1 SP

2 SP

4 SP

2 SP

6 SP

6 SP

9 SP

6 SP

6 SP

8 SP

6 SP

9 SP

2 SP

3 SP

6 SP 1

2

4

6

5

3High = 3 n

Medium = 1 n

Low = ½ n

13


Risk analysis were done, but the risks where prioritised using MoSCoW, which doesn’t automatically fit in a Agile environment.

Testing is prioritized based on product risks

=

M

S

W

C=

=

M

S

W

C =

M

S

W

C

Sprint 1 Sprint 2 Sprint 3

12


Agile Testing

isn’t Risking IT!

Introduction


Which problems did

I face?

How to do it right?

Recap-

Q & A

12345

6


Gather Risk Items• Gather & Order Requirements

• Gather Product Risks

Plot Risk Items• Map Requirements

& Product Risks

Calculate Risk Items• Determine Impact

& Likelihood Factors

• Score & Calculate Risk Items/Factors

Strategy to tactic• Risk class tactic• Sprint tactic

Gather Risk Items

14


Gather & Order Requirements

Rank #

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

REQ04

REQ02

REQ07

REQ03

REQ01

REQ05

REQ06

REQ08 REQ

09 REQ10

REQ14

REQ11

REQ12

REQ13

REQ15

H

M

LREQ15

REQ14

REQ13

REQ12

REQ11

REQ10

REQ09

REQ08

REQ07

REQ06

REQ05

REQ04

REQ03

REQ02

REQ01

REQ15

REQ14

REQ13

REQ12

REQ11

REQ10

REQ09

REQ08

REQ07

REQ06

REQ05

REQ04

REQ03

REQ02

REQ01

MU

ST

SH

OU

LD

CO

ULD

17


Gather Product Risks

PR 02

PR 05

PR 01

PR 04

PR 03

PR 08

PR 07

PR 06

PR 09

16


Gather Risk Items• Gather & Order

Requirements• Gather Product

Risks

Plot Risk Items• Map Requirements & Product Risks





Plot Risk Items

17


Requirements

Map Requirements & Product Risks 1/3

Product Risks

REQ04

PR02

PR05

PR01

PR04

PR03

PR08

PR07

PR06

REQ02

REQ07

REQ03

REQ01

REQ05

REQ06

REQ08 PR

09REQ09 REQ

10

REQ14

REQ11

REQ12

REQ13

REQ15


REQ14

REQ11

REQ12

REQ13

REQ15

Requirements


Product Risks

REQ04

PR02

PR05

PR01

PR04

PR03

PR08

PR07

PR06

REQ02

REQ07

REQ03

REQ01

REQ05

REQ06

REQ08 PR

09REQ09 REQ

10



20

Product Risk

Requirement

PR 01

PR02

PR03

PR04

PR05

PR06

PR07

PR08

PR09

REQ 01 ● ● ● ●

REQ 02 ● ● ●

REQ 03 ●

REQ 04 ● ●

REQ 05 ● ●

REQ 06 ● ●

REQ 07 ● ●

REQ 08 ● ●

REQ 09 ● ●

REQ 10 ● ● ●

REQ 11 ●

REQ 12 ●

REQ 13 ● ● ●

REQ 14 ● ● ●

REQ 15 ● ●

H

M

L

MU

ST

SH

OU

LD

CO

ULD




Risks


& Product Risks





Calculate Risk Items

21


Impact factors e.g.• Frequency of Use• Visibility• Cost of failure• …

Determine Impact & Likelihood Factors

Likelihood Factors e.g.• Complexity• Re-use• Interfaces• Dependencies• Size• …

Interfaces

None

1 - 2

> 2

Visibility

Internal

Partners

Public


Product Risk

Requirement

PR 01

PR02

PR03

PR04

PR05

PR06

PR07

PR08

PR09

REQ 01 ● ● ● ●

REQ 02 ● ● ●

REQ 03 ●

REQ 04 ● ●

REQ 05 ● ●

REQ 06 ● ●

REQ 07 ● ●

REQ 08 ● ●

REQ 09 ● ●

REQ 10 ● ● ●

REQ 11 ●

REQ 12 ●

REQ 13 ● ● ●

REQ 14 ● ● ●

REQ 15 ● ●

Score & Calculate Item/Factor

23

LIKELIHOOD PRODUCTRISK

RISKCLASS

IMPACT

Interfaces

Size Re-use TOTAL TOTAL Users Visibility

Cost of Failure

> 2 > 5 SP None High PR01

Medium Low < 5 Internal < 1k

None 2 – 5 SP Partly Low PR02

Medium Medium > 50 Public < 1k

1 – 2 2 – 5 SP Completely

Medium PR03

Medium Medium 5 - 50 Internal < 1k

2 > 5 SP Partly High PR04

High Medium > 50 Partners

> 10k

2 < 2 SP Completely

Medium PR05

Medium Medium < 5 Partners

< 1k

1 – 2 < 2 SP Completely

Low PR06

Low Low < 5 Internal 1k – 10k

1 – 2 2 – 5 SP Partly Medium PR07

High High 5 - 50 Public > 10k

None 2 – 5 SP None Low PR08

Low Low 5 – 50 Internal < 1k

None < 2 SP Completely

Low PR09

Low Low 5 - 50 Internal < 1k

PR01

PR02

PR03

PR04

PR05

PR06

PR07

PR08

PR09

PR 01

PR 02

PR 03

PR 04

PR 05

PR 06

PR 07

PR 08

PR 09

Product Risk

Requirement

PR 01

PR02

PR03

PR04

PR05

PR06

PR07

PR08

PR09

REQ 01 ? ? ? ?

REQ 02 ? ? ?

REQ 03 ?

REQ 04 ? ?

REQ 05 ? ?

REQ 06 ? ?

REQ 07 ? ?

REQ 08 ? ?

REQ 09 ? ?

REQ 10 ? ? ?

REQ 11 ?

REQ 12 ?

REQ 13 ? ? ?

REQ 14 ? ? ?

REQ 15 ? ?

HMMH

H

H

Product Risk

Requirement

PRX

PRX

PRX

REQ X H M M

REQ X H M L

REQ X M M L

M H M MM M H M M M H M M H M M H L M L M M M H L M L L L LM L




Risks


& Product Risks





Strategy to tactic

24


High Medium Low

Check documentation Inspection Walkthrough Peer review

UT test case review Test lead Tester -

ST test case review Test lead Tester -

Include test case in regression set

Sprint + release

Release -

Automate test case CodedUI Record & Playback

-

Test execution on browsers

Top 3 Top 2 Top 1

… … … …

Risk class tactic

High = Effort * 3 Medium = Effort * 2 Low = Effort * ½

25


Sprint Tactic – Scope Sprint 1

26

REQ15

REQ14

REQ11

REQ10

REQ01

REQ09

REQ07

REQ06

REQ02

REQ08

REQ04

REQ03

REQ13

REQ12

REQ05

PR09

PR08

PR07

PR06

PR05

PR04

PR03

PR02

PR01

Product Risk

Requirement

PR02

PR04

PR06

PR08

PR09

REQ 05 H MREQ 12 L REQ 13 M L L

Rank #

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15


Sprint Tactic – Scope Sprint 2

27

REQ15

REQ14

REQ11

REQ10

REQ01

REQ09

REQ07

REQ06

REQ02

REQ08

REQ04

REQ03

REQ13

REQ12

REQ05

PR09

PR08

PR07

PR06

PR05

PR04

PR03

PR02

PR01

Product Risk

Requirement

PR02

PR03

PR04

PR09

REQ 03 M REQ 04 M M REQ 08 H L

Rank #

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

© All rights reserved 28

Scrum bord

To Do Doing Done

REQ 3

REQ 4

REQ 8

S P R I N T 2


Insight into Product Risk & Requirement coverage

29

Product Risk

Requirement

PR01

PR02

PR03

PR04

PR05

PR06

PR07

PR08

PR09

REQ 05 REQ 12 REQ 13 REQ 03 M REQ 04 M M REQ 08 H LREQ 02 M M H REQ 06 M H REQ 07 M M REQ 09 M L REQ 01 M H M MREQ 10 M M M REQ 11 H REQ 14 L LREQ 15 M L

Doin

gTo

do

Product Risk

Requirement

PR01

PR02

PR03

PR04

PR05

PR06

PR07

PR08

PR09

REQ 05 REQ 12 REQ 13 REQ 03 REQ 04 REQ 08 REQ 02 M M H REQ 06 M H REQ 07 M M REQ 09 M L REQ 01 M H M MREQ 10 M M M REQ 11 H REQ 14 L LREQ 15 M L

Don

eD

oin

gTo

doProduct Risk

Requirement

PR01

PR02

PR03

PR04

PR05

PR06

PR07

PR08

PR09

REQ 05 REQ 12 REQ 13 REQ 03 REQ 04 REQ 08 REQ 02 REQ 06 REQ 07 REQ 09 REQ 01 M H M MREQ 10 M M M REQ 11 H REQ 14 L LREQ 15 M L

Don

eD

oin

gTo

do

Product Risk

Requirement

PR01

PR02

PR03

PR04

PR05

PR06

PR07

PR08

PR09

REQ 05 REQ 12 REQ 13 REQ 03 REQ 04 REQ 08 REQ 02 REQ 06 REQ 07 REQ 09 REQ 01 REQ 10 REQ 11 REQ 14 L LREQ 15 M L

Don

eD

oin

gTo

do

Product Risk

Requirement

PR01

PR02

PR03

PR04

PR05

PR06

PR07

PR08

PR09

REQ 05 REQ 12 REQ 13 REQ 03 REQ 04 REQ 08 REQ 02 REQ 06 REQ 07 REQ 09 REQ 01 REQ 10 REQ 11 REQ 14 REQ 15

Don

eD

oin

g


Recap

Agile RRBT• Gather Risk Items• Plot relationship between Risk Items• Calculate and Classify Risk Items• Differentiate the work

Lessons Learned• Analyses can only be done with the

backlog in place.• Plan to iterate and evolve your

analyses in sync with the sprints.• Don’t use the backlog as your only

starting point for your analyses.• The result of your analyses is essential

in the planning meeting.• Order your testing effort in the small.

30

B.W.F.P.M. BRONNEBERGTEST MANAGER / CONSULTANT

REQUIREMENT & QUALITY MANAGEMENT

[email protected]

WWW.BRAMBRONNEBERG.NL

@BRAMBRONNEBERG

+31 6 55 11 35 10

Documents

Agile Testing isn’t Risking IT!