SBET on Agile Projects v3 - stickyminds.com · Session Based Exploratory Testing on Agile Projects Lessons Learned at iContactfrom 2009 -2012 ... Charters and SBET were our primary

T4 Class

4/19/2012 9:45:00 AM

"Session-based Exploratory Testing

on Agile Projects"

Presented by:

Bob Galen

Deutsche Bank

Brought to you by:

340 Corporate Way, Suite 300, Orange Park, FL 32073

888-268-8770 ∙ 904-278-0524 ∙ [email protected] ∙ www.sqe.com

Robert ‘Bob’ Galen

Deutsche Bank Global Technology

Bob Galen is an agile coach at Deutsche Bank Global Technology and director, agile solutions at Zenergy Technologies, a North Carolina-based firm specializing in agile testing and leading agile adoption initiatives. Bob regularly speaks at international conferences and professional groups on topics related to software development, project management, software testing, and team leadership. He is a Certified Scrum Master Practicing (CSP), Certified Scrum Product Owner (CSPO), and an active member of the Agile Alliance and Scrum Alliance. In 2009, Bob published Scrum Product Ownership–Balancing Value from the Inside Out, which addresses the gap in guidance toward effective agile product management. You can reach Bob at [email protected] or [email protected]

1

Session Based Exploratory Testing on

Agile ProjectsLessons Learned at iContact from 2009 - 2012

Bob GalenVP & Agile Coach, Deutsche Bank

Global Technology

President & Principal Consultant

RGCG, LLC

[email protected]

Copyright © 2012 RGCG, LLC 2

Introduction

Bob Galen

� Somewhere ‘north’ of 20 years experience ☺

� Various lifecycles – Waterfall variants, RUP, Agile, Chaos, etc.

� Various domains – SaaS, Medical, Financial, Computer Systems, and Telecommunications

� Developer first, then Project Management / Leadership, then Testing

� ‘Pieces’ of Scrum in late 90’s; before Agile was Agile

� Agility @ Lucent in 2000 – 2001 using Extreme Programming

� Formally using Scrum since 2000

� Currently Agile Coach at Deutsche Bank Global Technologies in Cary, NC

� Connect w/ me via LinkedIn if you wish9

Bias Disclaimer:

Agile is THE BEST Methodology for Software Development�

However, NOT a Silver Bullet!

2


Outline

� Introduction – Exploratory Testing and SBET

� iContact Background

� History of our use of SBET

� Some key pro/con of our evolution

� Release mapping

� Q&A


Exploratory Testing

Basics

� What is Exploratory Testing—

� Basic dynamics?

� Principles?

� Practices?

� How do you prepare?

� How do you document the testing?

� Pairing?

� Agile implications?

3


Exploratory Testing

Session Based

� What are the dynamics surrounding sessions—

� What testing do you focus the sessions on?

� Time-boxes?

� Focus guidance – Charters?

� Guidelines?

� Documentation?

� De-brief, discovery sharing?

� Agile Implications?


Why is Session Based Exploratory Testing

(SBET) so applicable to agile teams?

� We’ll explore the travels we had at iContact in implementing and executing Exploratory Testing, then wrapping it with Sessions

� Nimble – handling the most critical aspects of the day; changing scope & requirements are relatively easily handled

� Reactive – always testing the “most important thing”

� Collaborative – focus on people and skill rather than completely defined test cases; lightweight; pairing; fosters a whole-team view

� Resilient – to change, to operating conditions (in development environment, interfaces, databases), to what’s working and what’s not

� Simple & Fast – to ramp-up, to begin testing, to train new hires, to document

4


iContact

Background

� Company formed in late 2003, so approximately 9 years old

� SMB eMail Marketing

� Singular SaaS product; small vs. medium business delineation;

w/options

� LAMP technology stack; heavy PHP use in middle and lower tiers

� Added testers to the teams in early 2009

� So from 2003 – 2008, development-driven testing

� Scrum shop from late 2007

� Cross-functional agile teams introduced early in 2009

� Mid 2011- mature Scrum shop making monthly deployments of

customer value

� Early 2012 – Kanban shop focusing on Continuous Deployment


Initial steps…

Early 2009 – early 2010

� We started with a small, singular script “smoke test” that was being

run on a periodic basis

� Adding a few new, functional tests every sprint

� Both/all were manually executed

� Then we broke the smoke test up into Charters and began to run

SBET as a means of

� Distributing the testing beyond a lone tester, across teams, and to gain

transparency

� Isolating Charters towards specific functionally focused Scrum teams

(Create, Integrate, Track, Core, Contacts)

� Reduce execution time and foster test pairing

5


Follow-on steps…

Early 2010 – Late 2010

� Charters and SBET were our primary testing vehicle; that and

collaboratively driven functional testing within our scrum teams

� The # of Charters began to grow

� Aligning towards each product team

� As we added / extended functionality

� Continued to expand our coverage

� Physically grow as a company

� We had always been a “paired” testing environment


Paired SBET

� From the beginning we felt that Exploratory Testing should be a

‘paired’ activity

� Developer + Developer

� Tester + Tester

� Developer + Tester

� Team member + Customer Support member

� Cross-team implications

� As we continued to leverage continuous SBET, the paths got somewhat

‘old’; we needed new eyes & new pairs

� Teams tested each others work; ran each others Charters

� Side-effect that teams needed to x-communicate more; heavily Wiki

based

6


Paired SBET

Benefits

� The impact of inviting customer support (and other x-functional team

members) was HUGE

� In understanding customer usage

� In adding / changing Charters and focusing domain knowledge

� Developers increased their understanding & empathy for testing

� It’s Hard!

� It’s Long!

� We need to leverage our unit test investments

� It needed broader automation!


Early 2011 Evolution

� We’re de-emphasizing SBET as our “do all” testing method. Focus

it on:

� Smoke Tests, for example after a hot-fix

� Sprint-based “Test Fests” that are whole-team based

� Risk-based regression testing

� Starting to write our test cases in Cucumber step format; so that we

can easily convert them / run them as Cucumber tests

� So in each sprint we develop functional tests AND adjust Charters for SBET as

appropriate

� We do a Test Plan per team that shares / exposes the Sprint x Sprint and

Release level testing strategy

� Building a repository of these functional tests

7


Late 2011 - 2012 Latest Evolution

� Converting our Charters to functional automation

� We’ve used the expanded Charters as a base for developing mostly Cucumber

based automation

� Found the expanded format helpful for the conversion

� Reducing the # of steps and information in the Charters

� Converting them back to a truer form of SBET Charter with little “leading”

information

� Couple with our Test Idea brainstorming / design sessions to drive more new

tests

� Re-analyzing some of our results; leading towards updates of our

testing approaches & artifacts

General Benefits of

SBET

� It was a quick way to begin testing in an agile context

� From scratch; w/o any formal test repository / base

� Inherently collaborative

� Inherently agile, context-based, risk-based; testing the most important

things of the moment

� Pairing worked wonders to foster a “Everyone Owns Quality/Testing”

mindset

� Empathy for testing challenges

� Instilled the right behaviors

� Pulled the client / customer facing folks into our teams


8

Regression Dynamics of

SBET

� Two-edged sword when using it for Regression

� On the one hand, the Charters get beefier and more prescriptive

� Logging looks very similar code drop over code drop

� Can manage, report, and control x-team execution

� But

� It’s really not the core intent of ET – we’re controlling the exploration

� No matter what you do, it’s not “repeatable”

� Can de-rail your Endgame from a defect discovery perspective

� In hindsight, probably not the best use


Planning & Reporting

� Each team is responsible for planning their Charters for a given set

of sessions.

� Planning occurs within Sprint Planning

� Testers define a Test Plan per sprint – point is to drive strategic thinking

within the team

� Identify and x-team dependencies for testing

� Test Fests per sprint & per release

� Clearly defined strategies (why) behind the testing focus is team-driven

� Results simply visible on the wiki as charters / sessions are completed

� War room to coordinate items found, questions, sharing, issues, etc.


9


The Agile Release Train

Synchronized

Iterate

Iterate

Team 1

Team 2

Team 3

Team 4

Iterate

Iterate

Harden Iterate Iterate Iterate

X-team

Harden

Harden

Harden

Harden

Iterate Iterate

Iterate Iterate

Iterate Iterate

Iterate Iterate Iterate Iterate

Iterate

Iterate

Iterate

Internal

Release

External

Release

Docs,

Training,

Support,

UAT,

Comp.

Team n

�

Continuous Integration





The Agile Release Train

Example: iContact / SaaS Model

Team 1

Team 2

Team 3

Team 4

Iterate

Iterate

Harden

X-team

Harden

Docs,

Training

Harden

Harden

Harden

Iterate

Iterate

Production

Release

Team 10�





3 weeks / 15 days 4-5 days

Rinse

&

Repeat

Environment

Evolution Dev + QA QA -> Staging Production

SBET, Exploratory –

Regression Testing

10

Retrospectives

� Leverage the teams’ Retrospectives as a place to discuss:

� SBET – testing strategy

� Timing

� Coverage

� X-team interactions

At a scrum team and release level.

� Reflect on our results and implications to

� Functional tests (additions, retiring, new suites, etc.)

� Extending our Charters


Test Ideas

� We’ve recently begun to use Rob Sabourin’s views towards Test

Idea generation - http://www.amibugshare.com/articles/

� Brainstorming

� Formation

� Prioritization

� Execution & Retrospective

� Very useful to dovetail the ideas to the execution within Charters

� Sort of an idea laboratory for your ideas

� Fast feedback for fine-tuning


11

Wiki-based examples of our

SBET artifacts

� We leverage a wiki (Confluence) for our SBET charter storage,

session notes, defects, and repetitive run logs

� We build in some forward learning notes, where applicable, for our

retrospectives

� So here are some examples9


Overall Charter - Findings

Each charter has a wiki page

associated with tracking results.

Results would be captured per session

including very specific bugs, notes,

and concerns / questions.


12

Charter Team Assignments

Team and individual assignments would

be made on a TestFest or event basis.

This page identified the notion of:

“Charter Owners” and identified groups,

individuals, pairs, that were

recommended to execute the charters.

This varied as our application risk-based

test planning drove us in different

directions. So very dynamic; but, with

some solid core of team familiarity per

charter�


Charter Test Index / Management

This wiki page rolls up all charters for overall test

execution.

We also used it to monitor execution and connect

to individual results.

You can see notes regarding “Automation

Assisted”. As we developed functional and other

automation, we “connected” it to the charters &

sessions.

So automation execution became “part of” the

sessions. Not so to de-focus the exploration, but

as a means of augmenting the focus and round-

trip evolution of automation on�

Charter-based boundaries.


13

Individual Charter - Logging

Each charter has a wiki page associated with

tracking results. Results would be captured per

session including:

� Software version, release target

� Who executed the charter

� Bugs found

� Script of “travels” in executing this session and

specific guidance around

� (1) how to reproduce bugs and

� (2) what would be good to convert into

functional test cases.


Individual Charter - Logging

Each charter has a wiki page associated with tracking

results. Results would be captured per session

including:

� Another important aspect of our logging was that all

history of execution was in a single wiki page. Yes,

this could get unwieldy over time, but we didn’t find

that to be the case�yet

� What we found is that sessions could be “directed”

based on historical notes on

� Stability vs. flakiness

� Previous “coverage”

� Previous “discoveries”


14

Individual Charter – Expanded Steps

Each charter has a wiki page associated with

tracking results. Results would be captured per

session including:

� Testing guidance including entry, exit

conditions and test steps

� We expanded on the steps NOT to inhibit

exploration,

� But as an effort to allow the charters and

execution notes to allow us to develop more

formal functional test cases

� That would be “extracted” from the SBET

efforts�


The Future – Next Steps

� We’re moving back to the traditional “exploratory” nature

of ET and Session-Based ET

� That and team-based Test Idea brainstorming and

� Agile test planning

� Are driving our testing creativity and value-added tests

� Kanban evolution – continuous flow

� Continuous Integration – @ every check-in

� Off of Trunk and notion of Feature Branches (MMF)

� Continuous Deployment – weekly deployments to

production


15

The Future – Next Steps

� We’re realizing our test

automation strategy

and gaining significant

automation

coverage9

� Allowing to next focus

on agile performance

testing, UX integration,

and continuously

improving tests


Wrapping up…

� SBET can be a very effective agile approach

� Beginning / Building team new to agile

� Ongoing team maturity

� Be careful not to ‘overload’ your Charters; allow/support for true

‘exploration’

� Pairing can be powerful

� Create wiki-based, very simple management & tracking

� Leverage Charters for your automation development

� Prioritization & experimentation

� Driving time savings & risk mitigation


16

Contact InfoBob Galen

Principal Consultant, RGalen Consulting Group, L.L.C.

Director of Agile Solutions, Zenergy Technologies,

Director, Agile Practices, iContact

Experience-driven agilefocused training, coaching

& consulting

Cell: (919) [email protected]

[email protected]

Blogs:

Project Times - http://www.projecttimes.com/robert-galen/

Business Analyst – BA Times -http://www.batimes.com/robert-galen/

My Podcast on all things ‘agile’ - http://www.meta-cast.com/

Scrum Product Ownership – Balancing Value From the Inside Out published by RGCG in 2009.


Documents

SBET on Agile Projects v3 - stickyminds.com · Session Based Exploratory Testing on Agile Projects Lessons Learned at iContactfrom 2009 -2012 ... Charters and SBET were our primary