Between Scrum and Kanban - define a test process for Agile methodologies

© 2010 Cognifide Limited. In commercial confidence only.© 2010 Cognifide Limited. In commercial confidence only.

Between Scrum and Kanban – define test process

Zbyszek Moćkun

© 2010 Cognifide Limited. In commercial confidence only.© 2010 Cognifide Limited. In commercial confidence only.

1. Methodologies

2. Problems to overcome

3. Workflows

4. Issue management

5. Taskboard

6. Exploratory testing

7. Metrics

Agenda

Client/partner logo

© 2010 Cognifide Limited. In commercial confidence only.© 2010 Cognifide Limited. In commercial confidence only.

Few words about Agile

Methodologies

© 2010 Cognifide Limited. In commercial confidence only.© 2010 Cognifide Limited. In commercial confidence only.

Methodologies axis

(120+)

RUP (13)

XP

(9)

Scrum

(3)

Kanban

(0)

Do whatever

Source: Kanban vs Scrum – how to make the best of both (Henrik Kniberg)

Prescriptive Adaptive

© 2010 Cognifide Limited. In commercial confidence only.© 2010 Cognifide Limited. In commercial confidence only.

Kanban

• Coverage

• Do not duplicate tests during regression

• Test specific functionality

• Visualize the workflow

• Limit WIP

• Measure the lead time

• Manage the lenght of the queue

• No fixed timeboxes iteration

• You can choose when:− Planning

− Release

− Retrospective

© 2010 Cognifide Limited. In commercial confidence only.© 2010 Cognifide Limited. In commercial confidence only.

Scrum-inspired

• Coverage

• Do not duplicate tests during regression

• Test specific functionality

• Do not limit yourself to one tool

• Mix and match tools as you need

• Dynamic changes in projects (suite your process to actual needs)

• Retrospective – continuous improvements

• Agile manifesto (Individuals and interactions over processes and tools)

© 2010 Cognifide Limited. In commercial confidence only.© 2010 Cognifide Limited. In commercial confidence only.

Problems that we had to overcome

Problems

© 2010 Cognifide Limited. In commercial confidence only.© 2010 Cognifide Limited. In commercial confidence only.

Problems to be overcame

• Coverage

• Do not duplicate tests during regression

• Test specific functionality

• Project status on Taskboard− Status of all tasks visualized on Taskboard

− No need of additional reports

− Visualize QA activities

• Exploratory testing management

• Ensure that all tasks are tested

• Metrics: quality, testing velocity, ...

• One process for different methodologies: Scrum, Kanban, mixed

© 2010 Cognifide Limited. In commercial confidence only.© 2010 Cognifide Limited. In commercial confidence only.

Ensure a QA place in workflow

Workflows

© 2010 Cognifide Limited. In commercial confidence only.© 2010 Cognifide Limited. In commercial confidence only.

The meaning of „Done”

• When do we finish our work?

• What is the latest action?

• Who is responsible for the transfer to the state done?

• What is the client role?

• Visualize the done state

• Done for Scrum vs Done for Kanban

© 2010 Cognifide Limited. In commercial confidence only.© 2010 Cognifide Limited. In commercial confidence only.

QA States

There are no QA states in standard workflows • assumption - close action is for client (sign off)• Green states

Add qa states to your workflow• QA queue, Testing in progress, Tested• Green plus white states

In Progress Resolved Testing in

Progress Tested Closed

© 2010 Cognifide Limited. In commercial confidence only.© 2010 Cognifide Limited. In commercial confidence only.

Keep the standards

• Do not change the workflow parts that customer use

In Progress Resolved Testing in

Progress Tested Closed

In Progress Implemented Testing in Progress Resolved Closed

© 2010 Cognifide Limited. In commercial confidence only.© 2010 Cognifide Limited. In commercial confidence only.

Different tasks, different needs

• Client oriented – extended workflow− QA Engineer is not always a client− Client has to accept task solution

• Single role – simple workflow− Division of task on the parts by developer

© 2010 Cognifide Limited. In commercial confidence only.© 2010 Cognifide Limited. In commercial confidence only.

Make it simple

Issues management

© 2010 Cognifide Limited. In commercial confidence only.© 2010 Cognifide Limited. In commercial confidence only.

Issues by software lifecycle

© 2010 Cognifide Limited. In commercial confidence only.© 2010 Cognifide Limited. In commercial confidence only.

Issues tracking

• Regression – Standard Issue Type

• Acceptance – Sub-Task Issue Type

© 2010 Cognifide Limited. In commercial confidence only.© 2010 Cognifide Limited. In commercial confidence only.

Story as a container

• All issues connected with story are reported as sub task (bug, improvement, question, task)

• All data in one place (historical data too)• All sub tasks have to be implemented to send story to QA• Story accepted only if no sub-issues left

Story

Sub improvment Sub bug Sub bug Sub bug Subtask

© 2010 Cognifide Limited. In commercial confidence only.© 2010 Cognifide Limited. In commercial confidence only.

Dev – QA cooperation

© 2010 Cognifide Limited. In commercial confidence only.© 2010 Cognifide Limited. In commercial confidence only.

Quality levels

• Are all issues have to be resolved?

• Can we fix some of them later?

• Flexible or restricted process?

© 2010 Cognifide Limited. In commercial confidence only.© 2010 Cognifide Limited. In commercial confidence only.

Retest: whole story or sub-issues only

• Switching between contexts require very good time managment skills

• Helicopter view• Do not duplicate data (JIRA <->TCM)• All failed issues attached to story• QA Engineer can make decision

− Retest whole story− Only failed issues− Chosen parts (change propagation)− Decision made on available data (fixed issues, introduced

changes, developer output)

© 2010 Cognifide Limited. In commercial confidence only.© 2010 Cognifide Limited. In commercial confidence only.

Integrate Workflow with Taskboard

Taskboard

© 2010 Cognifide Limited. In commercial confidence only.© 2010 Cognifide Limited. In commercial confidence only.

Workflow on Taskboard

• One tool, all information, no additional reports

• Only the Standard Issue Type visible− Acceptance issues are hidden

• Sorted by rank (priority)

• Rejected issues again in To Do queue

© 2010 Cognifide Limited. In commercial confidence only.© 2010 Cognifide Limited. In commercial confidence only.

Why scripted aproach doesn’t work

Exploratory testing

© 2010 Cognifide Limited. In commercial confidence only.© 2010 Cognifide Limited. In commercial confidence only.

Scripted testing doesn’t work

• Coverage

• Do not duplicate tests during regression

• Test specific functionality

• Status in two tools: JIRA and TCM

• Retesting dilemma

• Test Script creation in advance − boring and not efficient

− Can’t write scripts for 100% coverage

− Requirements/Acceptance criteria updates

− No time for this phase (parallel testing)

• Acceptance tests

© 2010 Cognifide Limited. In commercial confidence only.© 2010 Cognifide Limited. In commercial confidence only.

Scripting doesn’t work - reports

• Scripted testing report

• Not all founded issues are against test case

• Test case number means nothing (James Bach)

• New test scripts added almost each day

• Old onces are updated

phase All Executed passed failed

Test design phase 80 0 0 0

First day 90 (4 updated) 20 16 4

Second day 95 (10) 75 65 10

Third day 120 (14) 75 73 2

Fourth day 125 (16) 110 98 12

© 2010 Cognifide Limited. In commercial confidence only.© 2010 Cognifide Limited. In commercial confidence only.

Exploratory testing

• Efficiency− Concentrate on functionality

− Time management (quality level)

− Critical issues are founded earlier

− Changes are welcome

− No test script creation in advance phase

− Test scripts as output (for regression tests purpose)

• Learning phase− Simulate users (usability issues)

• Move responsibility on tester− Retesting dilemmas

© 2010 Cognifide Limited. In commercial confidence only.© 2010 Cognifide Limited. In commercial confidence only.

Improve your ET skills

© 2010 Cognifide Limited. In commercial confidence only.© 2010 Cognifide Limited. In commercial confidence only.

Find what you need

Metrics

© 2010 Cognifide Limited. In commercial confidence only.© 2010 Cognifide Limited. In commercial confidence only.

Metric by actions

© 2010 Cognifide Limited. In commercial confidence only.© 2010 Cognifide Limited. In commercial confidence only.

One measurement, two metrics

• Acceptance rate− Acceptance actions/all qa actions

− All QA actions = accept or reject

− Measure quality of software send to QA

• QA velocity− Sum off all actions

− Divide by number of QA Engineers assigned to project

− QA velocity depend on software quality that comes to qa

− Why testing takes so long? (Kaner’s blog)