Product Risk Analysis Workshop

© CGI Group Inc. 2013

Product Risk Analysis Workshop

Chris Schotanus

Contents of the Workshop

The theory of Product Risk Analysis

Workshop

Report Back

2

© Maarten Toonder Studio’s

http://images.google.com/imgres?imgurl=http://praw.nl/images/professor.gif&imgrefurl=http://praw.nl/&h=159&w=150&sz=5&hl=nl&start=3&um=1&tbnid=QFWFPP15osiZ2M:&tbnh=97&tbnw=92&prev=/images?q=Prlwytzkofsky&ndsp=20&svnum=10&um=1&hl=nl&rls=com.microsoft:en-US&sa=N

Risk & Requirement Based Testing

Risk & Requirement Based Testing

Risk and requirements analysis are two, independent activities

• Risk analysis:

• Identified by “all” stakeholders

• Analysis byond defined features and functions

• Requirements analysis

• Functional

• Non-functional

• Risk & Requirements Matching leads to a set of requirements that

• Have a functional priority

• Are linked to product risks and

• Have a risk priority

4

Risk Analysis

PREPARATION EXECUTION

Logica ©

Evaluation

and transfer

Issue

management

Progress

management

Reporting

and advice

Test

organisation

Estimation

Risk

analysis and

Test strategy

Planning

Risk and

requirement

based testing

Risk

analysis and

Test Strategy

5

Project versus product risks

6


7


8


9


10

Project Risk versus Product Risk

11

Project risks

Project overrun in time & budget

Lack of system standards

Insufficient resources

Unclear requirements

Incomplete documentation

mitigation measures

in the test plan

Product risks

Unsatisfactory quality

Incorrect functionality

Not user-friendly

Difficult to maintain

Low efficiency

Difficult to install

Test strategy based

on product risks

Risk Prioritization

Relative Priority

Compare:

Mobile phone backlight malfunction

versus

Space shuttle fuel system malfunction

13

http://www.google.nl/url?sa=i&rct=j&q=&esrc=s&frm=1&source=images&cd=&cad=rja&docid=yjdGl4vsjDaKeM&tbnid=5xrvhWzVjdvokM:&ved=0CAUQjRw&url=http://www.telecomnieuws.com/eerste-gsm-telefoon-gesprek-al-weer-20-jaar-geleden/&ei=TZxKUuHqKMj70wWL2YHQBQ&bvm=bv.53371865,d.bGE&psig=AFQjCNEpaJ2HoXJGwD0MkiS1LE1omaRtww&ust=1380707749339660

MoSCoW: an example

14

MoSCoW: an example

If this

goes

wrong

it will

have

...

15

MoSCoW: an example

If this

goes

wrong

it will

have

...

…financial consequences

for our customers.

16

MoSCoW: an example

If this

goes

wrong

it will

have

...


for our customers.

All customers

17

MoSCoW: an example

If this

goes

wrong

it will

have

...


for our customers.

Must test All customers

18

MoSCoW: an example

If this

goes

wrong

it will

have

...


for our customers.


One customer

19

MoSCoW: an example

If this

goes

wrong

it will

have

...


for our customers.


One customer Should test

20

MoSCoW: an example

If this

goes

wrong

it will

have

...

… no financial

consequences

for our customers


for our customers.



21

MoSCoW: an example

If this

goes

wrong

it will

have

...

… no financial

consequences

for our customers


for our customers.



Could test

All customers

One customer

Should test

22

MoSCoW: an example

If this

goes

wrong

it will

have

...

… no financial

consequences

for our customers


for our customers.



… financial consequences

for our department.

No workaround

Workaround Won’t test

Could test

Could test

All customers

One customer

Should test

23

MoSCoW: an example

If this

goes

wrong

it will

have

...

… no financial

consequences

for our customers


for our customers.



… no financial

consequences

for our department. Won’t test

No workaround

Workaround

Could test

… financial consequences

for our department.

No workaround

Workaround Won’t test

Could test

Could test

All customers

One customer

Should test

24

Stakeholder analysis

Stakeholders

A stakeholder is a person or organization that has a legitimate

interest in a product or service

26

Stakeholders

A stakeholder is a person or organization that has a legitimate

interest in a product or service

Who is responsible?

Who has a problem when things go wrong?

Who needs the system for normal operations?

Who will use the system?

Who sets regulatory rules?

27

Stakeholders - Examples

• Legal authorities like AFM, DNB, ECB,

Represented by Compliance & Legal department

• Helpdesk

• Operations

• Application Management

• (End-) users

• Marketing

• System development

• Testers

…

28

Stakeholders & Quality Attributes

Information

system

Development

Application Management

End user

Audit

Helpdesk

29

Stakeholders & Quality Attributes

Information

system

Development

Application Management

End user

Audit

Helpdesk

Requirements

on usability

Requirements on

changeability

Requirements on

security

Requirements on

maintainability

Requirements on

analysability

30

Quality Attributes ISO9126

Quality Attributes

ISO 9126

Functionality

Suitability

Accuracy

Interoperability

Security

Functionality

Compliance

Reliability

Maturity

Fault tolerance

Recoverability

Reliability

Compliance

Usability

Understandability

Learnability

Operability

Attractiveness

Usability

Compliance

Efficiency

Time behaviour

Resource utilisation

Efficiency

Compliance

Maintainability

Analysability

Changeability

Stability

Testability

Maintainability

Compliance

Portability

Adaptability

Installability

Co-existence

Replaceability

Portability

Compliance

31

Quality Attributes

ISO 25010

Functionality

Suitability

Functional

appropriateness

Accuracy

Reliability

Maturity

Availability

Fault tolerance

Recoverability

Performance

Efficiency

Time behaviour

Resource

utilization

Operability

Appropriateness

recognisability

Ease of use

User error

Protection

User interface

Aesthetics

Technical

Learnability

Technical

Accessibility

Security

Confidentiality

Integrity

Non–repudiation

Accountability

Authenticity

Portability

Adaptability

Installability

Replaceability

Compatibility

Co-existence

Interoperability

Maintainability

Modularity

Reusability

Analyzability

Modifiability

Testability

Quality Attributes ISO 25010

32

The Product Risk Analysis Workshop

The domain and the question

• Financial organization selling mortgages

• Direct writer and via independent intermediates

• New mortgage proposal system

• Company Policy

• Raise of market share

• Low cost, high quality

• Turnover up 10%

• Operational excellence

The question is:

what are the risks related to an IT system

used within the mortgage domain

Stakeholder analysis

35

Stakeholders

Stakeholder Responsibility

Marketing & Sales Defines and sells mortgages

Application management Functional/technical management of the

system

Legal Represent the legal authorities

System development Develop and maintain the system

Tester Test the system

Operations Keeps the system up and running

Client Relations Represents the end-user community

PRAW Planning

Determine the type of PRA

In case of a PRAW:

• Plan the workshop • Requirements experts

• Stakeholders

• Scribe

• Collect documentation • High level definition study

• Business Requirements Specification*)

• Checklists and Rule sets • Customize to fit application type

*)Is used during matching phase, not before!

Test MAnager

Team

Scribe

Team

Team

Team

Test Manager

Elicitation

Matching

Planning

Logging

Balancing

Prioritization

Exit

PRAW Phases

37

Risk Elicitation

Brown Paper Session

A Brown Paper Session is a session where experiences,

needs and ideas from different points of view are

brought together (on large sheets of brown paper).

39

“Copafijth” aspects

• Communicatie (Communication)

• Organisatie (Organisation)

• Personeel (Staff)

• Administratieve Organisatie (Administrative Organisation)

• Financieel (Financial)

• Informatie (Information)

• Juridisch (Legal)

• Techniek (Technical)

• Huisvesting (Housing/logistics)

This list will support the risk elicitation process

40

COPAFIJTH is a Dutch acronym that indicates all aspect of an organisation

Risk Balancing

Assignation of priorities

dinsdag 1 oktober 2013

MoSCoW: an example

Must test

Should test

Could test

Won’t test

Won’t test

Could test

Could test

Should test

Non-financial impact

own department

Financial impact

own department

Non-financial

customer impact

Financial

customer impact

If this

failure

occurs,

it has

...

Workaround

No workaround

Workaround

No workaround

One customer

All customers

One customer

All customers

42

Report Back

Your experiences

43

The result

The result of a product risk analysis is an agreed and complete list

of product risks and their related priorities

44

The result

The result of a product risk analysis is an agreed and complete list

of product risks and their related priorities

45

But what’s the use?

First advandtage: Risk & Requirement Matching

Requirements

Product Risks Risk, no

requirement:

• Add requirement

• Leave out risk

Requirement, no

risk:

• Add risk and test

• Leave out requirement

Second advantage: Strategic Test Slicing Method

(STSM)

Won’t test

conditions:

• …

• …

Could test

conditions:

• …

• …

• …

• …

Should test

conditions:

• …

• …

• …

• …

• …

• ...

Must test

conditions:

• …

• …

• …

• …

OUCH!

Priorities in the test management process

48


49

Number of test cases

within available budget


50



Planning and

sequence


51



Planning and

sequence

Status in budget

and quality


52



Planning and

sequence

Status in budget

and quality

Which defect will

be solved and

which is postponed?


53



Planning and

sequence

Status in budget

and quality

Which defect will

be solved and

which is postponed?

Are all risks mitigated:

can we deploy?

Max 30 days

24 hours

Product Backlog

As prioritized by

Product Owner

Sprint Backlog

Backlog tasks

expanded

by team

Potentially Shippable

Product Increment

Daily Scrum

Meeting

Source: Adapted from Agile Software

Development with Scrum by Ken

Schwaber and Mike Beedle. Product Owner Scrum master Team member Stakeholder

Risk priorities in the development process

54

Product

Product risk

Requirement

PR

01

PR

02

PR

03

PR

04

PR

05

PR

06

PR

07

PR

08

PR

09

REQ 01 ● ● ● ●

REQ 02 ● ● ●

REQ 03 ●

REQ 04 ● ●

REQ 05 ● ●

REQ 06 ● ●

REQ 07 ● ●

REQ 08 ● ●

REQ 09 ● ●

REQ 10 ● ● ●

REQ 11 ●

REQ 12 ●

REQ 13 ● ● ●

REQ 14 ● ● ●

REQ 15 ● ●

Risk Matrix M

ust

Have

Sh

ou

ld H

ave

Wo

uld

Have

Must Test ShouldTest WouldTest

Risk Matrix

Product risk

Requirement

PR

01

PR

02

PR

03

PR

04

PR

05

PR

06

PR

07

PR

08

PR

09

REQ 01 ? ? ? ?

REQ 02 ? ? ?

REQ 03 ?

REQ 04 ? ?

REQ 05 ? ?

REQ 06 ? ?

REQ 07 ? ?

REQ 08 ? ?

REQ 09 ? ?

REQ 10 ? ? ?

REQ 11 ?

REQ 12 ?

REQ 13 ? ? ?

REQ 14 ? ? ?

REQ 15 ? ?

Mu

st

Have

Sh

ou

ld H

ave

Wo

uld

Have


Risk Matrix

Product risk

Requirement

PR

01

PR

02

PR

03

PR

04

PR

05

PR

06

PR

07

PR

08

PR

09

REQ 01 ? ? ? ?

REQ 02 ? ? ?

REQ 03 ?

REQ 04 ? ?

REQ 05 ? ?

REQ 06 ? ?

REQ 07 ? ?

REQ 08 ? ?

REQ 09 ? ?

REQ 10 ? ? ?

REQ 11 ?

REQ 12 ?

REQ 13 ? ? ?

REQ 14 ? ? ?

REQ 15 ? ?

Product risk

Requirement

PR

X

PR

X

PR

X

REQ X H M M

REQ X H M L

REQ X M M L

Mu

st

Have

Sh

ou

ld H

ave

Wo

uld

Have


Risk Matrix

Product risk

Requirement

PR

01

PR

02

PR

03

PR

04

PR

05

PR

06

PR

07

PR

08

PR

09

REQ 01 ? ? ? ?

REQ 02 ? ? ?

REQ 03 ?

REQ 04 ? ?

REQ 05 ? ?

REQ 06 ? ?

REQ 07 ? ?

REQ 08 ? ?

REQ 09 ? ?

REQ 10 ? ? ?

REQ 11 ?

REQ 12 ?

REQ 13 ? ? ?

REQ 14 ? ? ?

REQ 15 ? ?

Mu

st

Have

Sh

ou

ld H

ave

Wo

uld

Have


Risk Matrix

Product risk

Requirement

PR

01

PR

02

PR

03

PR

04

PR

05

PR

06

PR

07

PR

08

PR

09

REQ 01 ? ? ? ?

REQ 02 ? ? ?

REQ 03 ?

REQ 04 ? ?

REQ 05 ? ?

REQ 06 ? ?

REQ 07 ? ?

REQ 08 ? ?

REQ 09 ? ?

REQ 10 ? ? ?

REQ 11 ?

REQ 12 ?

REQ 13 ? ? ?

REQ 14 ? ? ?

REQ 15 ? ?

M

Mu

st

Have

Sh

ou

ld H

ave

Wo

uld

Have


Risk Matrix

Product risk

Requirement

PR

01

PR

02

PR

03

PR

04

PR

05

PR

06

PR

07

PR

08

PR

09

REQ 01 ? ? ? ?

REQ 02 ? ? ?

REQ 03 ?

REQ 04 ? ?

REQ 05 ? ?

REQ 06 ? ?

REQ 07 ? ?

REQ 08 ? ?

REQ 09 ? ?

REQ 10 ? ? ?

REQ 11 ?

REQ 12 ?

REQ 13 ? ? ?

REQ 14 ? ? ?

REQ 15 ? ?

M M

Mu

st

Have

Sh

ou

ld H

ave

Wo

uld

Have


Risk Matrix

Product risk

Requirement

PR

01

PR

02

PR

03

PR

04

PR

05

PR

06

PR

07

PR

08

PR

09

REQ 01 ? ? ? ?

REQ 02 ? ? ?

REQ 03 ?

REQ 04 ? ?

REQ 05 ? ?

REQ 06 ? ?

REQ 07 ? ?

REQ 08 ? ?

REQ 09 ? ?

REQ 10 ? ? ?

REQ 11 ?

REQ 12 ?

REQ 13 ? ? ?

REQ 14 ? ? ?

REQ 15 ? ?

H M M

Mu

st

Have

Sh

ou

ld H

ave

Wo

uld

Have


Risk Matrix

Product risk

Requirement

PR

01

PR

02

PR

03

PR

04

PR

05

PR

06

PR

07

PR

08

PR

09

REQ 01 ? ? ? ?

REQ 02 ? ? ?

REQ 03 ?

REQ 04 ? ?

REQ 05 ? ?

REQ 06 ? ?

REQ 07 ? ?

REQ 08 ? ?

REQ 09 ? ?

REQ 10 ? ? ?

REQ 11 ?

REQ 12 ?

REQ 13 ? ? ?

REQ 14 ? ? ?

REQ 15 ? ?

H M M

H

Mu

st

Have

Sh

ou

ld H

ave

Wo

uld

Have


Risk Matrix

Product risk

Requirement

PR

01

PR

02

PR

03

PR

04

PR

05

PR

06

PR

07

PR

08

PR

09

REQ 01 ? ? ? ?

REQ 02 ? ? ?

REQ 03 ?

REQ 04 ? ?

REQ 05 ? ?

REQ 06 ? ?

REQ 07 ? ?

REQ 08 ? ?

REQ 09 ? ?

REQ 10 ? ? ?

REQ 11 ?

REQ 12 ?

REQ 13 ? ? ?

REQ 14 ? ? ?

REQ 15 ? ?

H M M

H

H

Mu

st

Have

Sh

ou

ld H

ave

Wo

uld

Have


Risk Matrix

Product risk

Requirement

PR

01

PR

02

PR

03

PR

04

PR

05

PR

06

PR

07

PR

08

PR

09

REQ 01 ? ? ? ?

REQ 02 ? ? ?

REQ 03 ?

REQ 04 ? ?

REQ 05 ? ?

REQ 06 ? ?

REQ 07 ? ?

REQ 08 ? ?

REQ 09 ? ?

REQ 10 ? ? ?

REQ 11 ?

REQ 12 ?

REQ 13 ? ? ?

REQ 14 ? ? ?

REQ 15 ? ?

H M M

H

H

H

Mu

st

Have

Sh

ou

ld H

ave

Wo

uld

Have


Risk Matrix

Product risk

Requirement

PR

01

PR

02

PR

03

PR

04

PR

05

PR

06

PR

07

PR

08

PR

09

REQ 01 ? ? ? ?

REQ 02 ? ? ?

REQ 03 ?

REQ 04 ? ?

REQ 05 ? ?

REQ 06 ? ?

REQ 07 ? ?

REQ 08 ? ?

REQ 09 ? ?

REQ 10 ? ? ?

REQ 11 ?

REQ 12 ?

REQ 13 ? ? ?

REQ 14 ? ? ?

REQ 15 ? ?

H M M

H

H

H

M H M M

M M H

M

M M

H M

M H

M M

H L

M L

M M M

H

L

M L L

L L

M L

Mu

st

Have

Sh

ou

ld H

ave

Wo

uld

Have


Selecting from the product backlog

During the planning poker session the risk priorities are taken into

account

• Risks relate to story points

• Combining user stories efficiently

Sprint 1 Sprint 2 Sprint 3



account






account






account




Risk & Requirement Based Planning

Product risk

Requirement

PR

01

PR

02

PR

03

PR

04

PR

05

PR

06

PR

07

PR

08

PR

09

REQ 05

REQ 12

REQ 13

REQ 03 M

REQ 04 M M

REQ 08 H L

REQ 02 M M H

REQ 06 M H

REQ 07 M M

REQ 09 M L

REQ 01 M H M M

REQ 10 M M M

REQ 11 H

REQ 14 L L

REQ 15 M L


Mu

st

Have

Sh

ou

ld H

ave

Wo

uld

Have


Product risk

Requirement

PR

01

PR

02

PR

03

PR

04

PR

05

PR

06

PR

07

PR

08

PR

09

REQ 05

REQ 12

REQ 13

REQ 03

REQ 04

REQ 08

REQ 02 M M H

REQ 06 M H

REQ 07 M M

REQ 09 M L

REQ 01 M H M M

REQ 10 M M M

REQ 11 H

REQ 14 L L

REQ 15 M L


Mu

st

Have

Sh

ou

ld H

ave

Wo

uld

Have


Product risk

Requirement

PR

01

PR

02

PR

03

PR

04

PR

05

PR

06

PR

07

PR

08

PR

09

REQ 05

REQ 12

REQ 13

REQ 03

REQ 04

REQ 08

REQ 02

REQ 06

REQ 07

REQ 09

REQ 01 M H M M

REQ 10 M M M

REQ 11 H

REQ 14 L L

REQ 15 M L


Mu

st

Have

Sh

ou

ld H

ave

Wo

uld

Have


Product risk

Requirement

PR

01

PR

02

PR

03

PR

04

PR

05

PR

06

PR

07

PR

08

PR

09

REQ 05

REQ 12

REQ 13

REQ 03

REQ 04

REQ 08

REQ 02

REQ 06

REQ 07

REQ 09

REQ 01

REQ 10

REQ 11

REQ 14 L L

REQ 15 M L


Mu

st

Have

Sh

ou

ld H

ave

Wo

uld

Have


Product risk

Requirement

PR

01

PR

02

PR

03

PR

04

PR

05

PR

06

PR

07

PR

08

PR

09

REQ 05

REQ 12

REQ 13

REQ 03

REQ 04

REQ 08

REQ 02

REQ 06

REQ 07

REQ 09

REQ 01

REQ 10

REQ 11

REQ 14

REQ 15


Mu

st

Have

Sh

ou

ld H

ave

Wo

uld

Have

Risk & Requirement Based Reporting

Product risk

Requirement

PR

01

PR

02

PR

03

PR

04

PR

05

PR

06

PR

07

PR

08

PR

09

Product risk

Requirement

REQ 05

REQ 12

REQ 13

REQ 03

REQ 04

REQ 08

REQ 02

REQ 06

REQ 07

REQ 09

REQ 01

REQ 10

REQ 11

REQ 14

REQ 15

So, when do we use the PRA?

During the test process

• Planning/estimation

• Traceability

• Reporting

• Go/No-go decision

Within the (agile) development proces

• Planning/estimation

• Rankschikking of the backlog

• Progress

• Rapportage

76

Product risk

Requirement

Test condition

Test case

Defect

© CGI Group Inc. 2013

Evaluation

dinsdag 1 oktober 2013

Documents

Product Risk Analysis Workshop