Upload
letruc
View
215
Download
2
Embed Size (px)
Citation preview
2011
Testing in the Enterprise Using Scrum
QUEST Boston
Wednesday, April 6th, 2011
9:45 AM – 10:45 AM
PRESENTER:
Robert Galen
COMPANY:
RGCG, LLC
Testing in the Enterprise using SCRUM
Stretching Scrum to Accommodate Legacy & Large-Scale Testing Activity
Bob GalenDirector R&D, iContact
President & Principal Consultant RGCG, LLC
Copyright © 2011, RGCG, LLC 2
IntroductionBob Galen
Somewhere ‘north’ of 20 years experience ☺Various lifecycles – Waterfall variants, RUP, Agile, Chaos, etc.Various domains – SaaS, Medical, Financial, Computer Systems, and TelecommunicationsDeveloper first, then Project Management / Leadership, then Testing‘Pieces’ of Scrum in late 90’s; before Agile was AgileAgility @ Lucent in 2000 – 2001 using Extreme ProgrammingFormally using Scrum since 2000Currently at iContact as Director R&D and Agile Architect/CoachConnect w/ me via LinkedIn if you wish…
Bias Disclaimer:Agile is THE BEST Methodology for Software Development…
However, NOT a Silver Bullet!
Copyright © 2011, RGCG, LLC 3
Outline
Introduction1. Agile Scaling & Scrum2. The Role of the Product Owner in Scaling3. Scrum-in-Testing4. Challenges5. Q&A
Copyright © 2011, RGCG, LLC 4
Introduction
Copyright © 2011, RGCG, LLC 5
Aspects of Agile Scaling
There are really 4 aspects to Scaling within Agility:Organizational – larger teams, distributed teams, and outsourced teamsProduct – larger scale projects, interoperability, usability & consistency, and forecastingDevelopment – dependencies & integration, varying processes, and cross group collaborationTesting & Deployment – regression, regulatory, integration, product maturation visibility, and production deployment readiness
This presentation focuses on these areas from a testing perspective, but also intersects the other points
Of course, in a small, pure agile implementations, much of this is unnecessary and contrary to the basic principles of agility
Copyright © 2011, RGCG, LLC 6
Aspects of Agile Scaling
Industry lessons have lagged in larger scale Agile implementationsIt’s not the “sweet spot” for them and we seem to be mostly on our own
In the Enterprise, Scrum is leading the way in providing guidance towards scaling, but so far it’s sparse in nature –
Ken Schwaber – published Enterprise Scrum in 2007Dean Leffingwell – published Scaling Software Agility in 2007Jeff Sutherland has taken the lead in defining and sharing lessons learnedThere are still gaps from a Product Owner perspective – although the Certified Scrum PO class tries to help
Large-scale testing implications have been largely ignored to-date –thus this presentation…
Copyright © 2011, RGCG, LLC 7
Aspects of Agile ScalingScrum of Scrums (of Scrums)
Source: Mike Cohn’s www.mountaingoatsoftware.com website.
Meta Scrum Level – (S3)
Scrum of Scrums - (S2)
Scrum Team Level - (S1)
Copyright © 2011, RGCG, LLC 8
Aspects of Agile Scaling
And there are roles not defined behind the SoSoS, for example –What are the sorts of conversations & activities that occur at each level?
Who guides the process, tools, & techniques for consistency?
What are the hierarchies behind the SoSoSScrumMaster(s), Product Owner(s), Lines of businessTeam collaboration – resource management, allocation, and budgetingReporting & Release coordinationQuality, Measurement, and Governance
And how do they operate, integrate, and provide consistency w/o command-and-control?
Copyright © 2011, RGCG, LLC 9
Aspects of Agile Scaling Jeff Sutherland – Parallel Scrum
Sutherland is leading the way toward modifying Scrum towards greater efficiency
Scrum levels – A, B, and C
Sutherland has been using Type C Scrum for 3-4 years at PatientKeeper
Sort of the “Nirvana” Scrum state, anyone else at Type C?
The key point is the organizational change dynamicsSprint transition time compressionForward thinking towards staging & deliveryParallel --- Everything!Organization-wide change implications – including Testing & Quality
Copyright © 2011, RGCG, LLC 10
Scrum EvolutionParallel Pipelining
Type A ScrumIsolated cycles of work
Type B ScrumOverlapping IterationsBacklog continuously refreshedReduced staging times
Type C ScrumAll at once, multiple –simultaneous releasesOrganization of a Meta-Scrum
Copyright © 2011, RGCG, LLC 11
Evolving Role of the Product Owner
Copyright © 2011, RGCG, LLC 12
Product OwnersTheir Evolving Role
Within each Scrum, the PO is typically narrowly focused on crafting the Backlog, engaging in progress, and reviewing sprint results
However, as Scrum scales, the PO team needs to become more focused on:
Product Line Evolution – Meta Backlog and coordination across Sprint teams, strategy development & execution, resource load-balancing, and budgeting Cross-Team Planning – SoS coordination, linked goals and backlog work, delivery integration, and staged (forward-thinking) planningDelivery Dynamics – timing, marketing, packaging, interrelationships, customer feedback, and achieving production quality
All of course with the team(s) delivering the product
Copyright © 2011, RGCG, LLC 13
Product OwnersGuiding Testing
The PO also needs to have a Quality & Testing perspective that within each Sprint focuses on –
Developing specific multifaceted quality release goalsWorking with the team (Testers) to develop acceptance testsWorking with the team to ensure daily convergence towards goals
Across the SoS:Developing Product & Portfolio quality meta goals that cross all Sprints Integrating deliverables and qualifying the overall Product via planned Integration & Stabilization SprintsBalancing automation vs. manual testing capacity and investmentFocusing teams towards Product release points
Copyright © 2011, RGCG, LLC 14
Product OwnersCoordination Workflow
Scrum of Scrum of Scrums –The PO organization must coordinate
Feature timing & workflowQuality & integration workflowProduct maturation and release readinessProduction deployment
In conjunction with technology and project leadership
While often interfacing to operations and customer facing organizations
Product Families
Applets
Applications
Products
Stories
Features
DevelopmentWorkflow
Testing & Evolving
Maturation
ProductIntegration
Copyright © 2011, RGCG, LLC 15
Product OwnerPlanning Levels in Large-Scale Agile Projects
Release Plans
Iteration Plans
Daily Plans
Product Vision
Product Roadmap
Yearly by Product Owner (s)
Semi-Annually by the Product Owner (s)
Quarterly by the Product Owner & Team (s)
Monthly or bi-weekly by the Team (s)
Daily by the team members
SmallTo
Large
Copyright © 2011, RGCG, LLC 16
Release Train ManagementIterative model with a release target
Product centricFocused on a production push/release
Synchronized Sprints across teams
Some teams are un-synchronized, but leads to less efficient cross-team (product) interactions
Continuous Integration is the glue
Including automated unit and feature tests; partial regression
Notion of a “Hardening Sprint”Focused more on Integration & Regression testingAssumption that it’s mostly automatedEnvironment promotion
Define a final Hardening Sprint where the product is readied for release
Documentation, Support, Compliance, UAT, Training
Copyright © 2011, RGCG, LLC 17
The Agile Release TrainSynchronized
Iterate
Iterate
Team 1
Team 2
Team 3
Team 4
Iterate
Iterate
Harden Iterate Iterate Iterate
X-teamHarden
Harden
Harden
Harden
Iterate Iterate
Iterate Iterate
Iterate Iterate
Iterate Iterate Iterate Iterate
Iterate
Iterate
Iterate
Internal Release
External Release
Docs,Training,Support,
UAT,Comp.
Team n
…
Continuous Integration
Continuous Integration
Continuous Integration
Continuous Integration
Copyright © 2011, RGCG, LLC 18
Release Goal SettingA Key for Coordination
As you scale, each planning level should create criteria (SprintGoals) that are –
Interrelated and cohesiveFocused towards the end product release and not simply on each teams deliverablesIdentify dependencies and overall workflow
The traditional notion of Chartering also applies at the higher levels, with Charters defined as:
Goals, Objectives & ScopeClearly measurable view to “Done” – Release CriteriaMulti-faceted view towards quality (defects, coverage, non-functional requirements)Allowing for team based scope trade-offs
Copyright © 2011, RGCG, LLC 19
Scrum-in-Testing
Copyright © 2011, RGCG, LLC 20
Process Overview
30 days
24 hours
Product BacklogAs prioritized by Product Owner
Sprint Backlog
Backlog tasksexpandedby team
Potentially ShippableProduct Increment
Daily ScrumMeeting
Source: Adapted from Agile Software Development with Scrum by Ken Schwaber and Mike Beedle.
The Agile intent is to perform all testing within the iteration – usually via
automation.Unit & Acceptance are the typical focus, but
what about other forms of testing?
Legacy regression, interoperability across sprints, performance,
usability, etc. Thus the rub!
Copyright © 2011, RGCG, LLC 21
Inherently Narrow Focusof Agile Testing
Typical Agile Team Testing Focus
Unit TestingAutomated Builds –
Smoke TestingFocused – Customer Acceptance Testing
Test Driven Development (TDD)
Very limited – Integration & Regression Testing
Focused Towards Automation
Limited Exploratory Testing
What’s Missing for Larger Scale Organizations?
Integration TestingFunctional Testing
System TestingRegression Testing
Performance TestingLoad Testing
Scenario Based TestingUser Acceptance Testing (UAT)
Usability Testing, Other “ility” TestingExploratory Testing
Large-scale Automation
Copyright © 2011, RGCG, LLC 22
Inherently Narrow Focusof Agile Testing
Early as possible testing – unit level TDDCustomer engaged in defining acceptance criteria & testsEarly defect prevention, detection & early repairHigh degree of automation –“everything” runs within the iterationQuality as a team responsibilityOngoing refactoring as required
Think of Lean principles as an underlying driver…
Just Enough, Just in TimeDeliver as Fast as PossibleTeam integrity & professionalismHolistic, built-in quality
Copyright © 2011, RGCG, LLC 23
Scrum-in-Testing Transitional Drivers
High levels of traditional testing Manual regression burden; Legacy systems; Habits across the business stakeholder team; External pressures or expectationsPO requires this as part of the Backlog
Early Scrum implementationsDevelopment teams struggle with gaining testing traction – not meeting Agile code quality goals or core practicesTesting teams falling into traditional behaviors – artifact based, gatekeeper mentality, and low adaptability or flexibilityInsufficient testing resources to staff Sprints and post Development Sprint testing requirements (thinking that Agile teams inherently need less testers)
Copyright © 2011, RGCG, LLC 24
Scrum-in-Testing Coordination Drivers
IntegrationShort term integration across (many) Sprinting Development teamsIntegrating with external data providersIntegrating with 3’rd parties and outsourcersEquipment limitations & production deployment / promotion models
The need to give an external team insights into deliverables forincluding in future work
PO driven Portfolio & Product road-maps; PMO oriented planning
Regulatory or compliance requirements (traceability and/or artifact evidence)
Copyright © 2011, RGCG, LLC 25
Danger Will Robinson!!!Disclaimer
I’m going to discuss an approach that may be helpful in certain contexts
Larger-scale projects; with large legacy code bases, little – no test automation in place; Enterprise environments; Regulated environmentsThis s-t-r-e-t-c-h-e-s Agile / Scrum approaches; use with CARE!
These are extensions to ‘Pure’ ScrumThey’re intended to be a temporary extensionEnsure your Product Owner understands what’s done and not done!They’re inefficient from a LEAN perspectiveThey can be a slippery slope and lead to regression into waterfallThey are ScrumBut
Copyright © 2011, RGCG, LLC 26
Scrum-in-Testing Strategies
Extending testing activity within the Sprint to include as much coverage as possible:
Of course, unit & acceptance for the current iteration featuresRe-running existing unit & acceptance testsMaintaining existing automationRunning partial integration & partial regression testing as possibleCross-team collaboration
Which usually requires a different staffing mix for each Sprint
Or extending the iterative model to accommodate testing via –Stabilization Sprints (sometimes called Hardening or Release Sprints)Skewed Development & Testing focused Sprints
Copyright © 2011, RGCG, LLC 27
Scrum-in-Testing Stabilization Sprints
30 day
Dev. Sprint(s)
Variable length
Sprint(s)
Product Owner defined Product BacklogsAnd “coordinates” between development sprints
ProductionProduct Release
Stablilzation Sprint(s) – focused on integrating development release forward towards production release
30 day
Dev. Sprint(s)
Testing Activities:1. Full regression2. Overall integration3. Performance4. Usability5. Bug fixing6. Production –
promotion steps
Copyright © 2011, RGCG, LLC 28
Inherently Narrow Focusof Agile Testing
Early as possible testing – unit level TDDCustomer engaged in defining acceptance criteria & testsEarly defect prevention, detection & early repairHigh degree of automation –“everything” runs within the iterationQuality as a team responsibilityOngoing refactoring as required
Think of Lean principles as an underlying driver…
Just Enough, Just in TimeDeliver as Fast as PossibleTeam integrity & professionalismHolistic, built-in quality
Copyright © 2011, RGCG, LLC 29
Scrum-in-Testing Stabilization Sprints
This is a modified version of the Scrum model where Sprints evolve from ►►►
Note: iteration resource mix changes as you move from development towards stabilization
1. Pure Development focused Sprints –delivering features to PO to…
2. Early Integration focused Sprints –coordinating a building product story and shaking our interoperability dependencies to…
3. Pure Testing focused Sprints – performing more traditional testing activity leading towards production release.
4. Finally and potentially Testing Infrastructural focused Sprints – automation maintenance, next iteration readiness, and customer / usability collaboration
Copyright © 2011, RGCG, LLC 30
Scrum-in-TestingStabilization Sprint – Sample Workflow
4 Development Sprints
Normal team composition
Early transition to Stabilization Sprints
50/50 Development + Testing focus
2 Stabilization Sprints
Strongly connected to development, led by strong testing team
Next development Sprint beginnings –
Iteration #0, gaining traction, Sprint #1
Repeat…
There is a resource transition cycle from full Development Sprints forward to full Stabilization Sprints.
The “Art” is in effective collaboration, resource sharing & communication – forward & backward
Copyright © 2011, RGCG, LLC 31
Scrum-in-TestingStabilization Sprint – Content Pressure Release
Think of Stabilization Sprint(s) as a feature content pressure release mechanism. As your content increases, so does its integration risk. You’ll want to use them –
When you have many Development Sprints running in parallelAs an interim integration mechanismWhen Stabilization Sprint threads run in parallel it creates the need for S2 & S3 oriented interactions
The model typically is used for larger numbers of Development Sprints contributing to a large-scale enterprise product
Duration and focus can vary from one Stabilization Sprint to the next
Copyright © 2011, RGCG, LLC 32
Scrum-in-Testing Skewed Testing Sprints
(n) day Test
Sprint(s)
Product Owner & test team members coordinate bug feedback into current Development Sprint Backlog and Product Backlog
Interim or IntegrationProduct Releases
Skewed Testing Sprint(s) – focused on providing more formal testing feedback by virtually running development & testing in parallel / skewed Sprints
30 day
Dev. Sprint(s)
Testing Activities:1. Partial regression2. Limited integration3. Early performance4. Bug fixing5. QA – promotion
steps
Copyright © 2011, RGCG, LLC 33
Scrum-in-Testing Skewed Testing Sprints
This model balances some traditional testing post Development Sprint against bogging down the sprint w/too broad a level of testing
Usually the testing is focused towards traditional regression and integration testing
May also be used for performance, compliance and other non-functional testing activity
The model typically is used for domains with an existing large scale legacy testing burden (or requiring larger scale testing practices)
In this case, the skew allows the development activity to progress while testing is performedSometimes this is viewed as “Waterfall” testing within Scrum; but becomes necessary if you can’t perform ALL testing within the iteration
Copyright © 2011, RGCG, LLC 34
Scrum-in-Testing Skewed Testing Sprints
Advantage of handling defects and integration issues close to the originating Sprint
Can reserve Sprint Backlog time so that changes can be incorporated immediately in the “next” Development Sprint
Testing Sprints are usually staffed solely with testers
In later phases, the testing can turn into more of a Stabilization sprint effort – so a morphing of the two strategies
Over time as your Agile experience grows, you’ll want to narrow the skew – perhaps with overlapping iterations
Over-arching goal is to eliminate the need for them!
Copyright © 2011, RGCG, LLC 35
Scrum-in-Testing Challenges
Copyright © 2011, RGCG, LLC 36
Scrum-in-Testing Typical Challenges – Alignment
It’s important to try and align Scrum iteration tempo across your SoSoS Sprints, putting pressure on your:
Cross-sprint Meta Backlog management & planningSprint Review results & feedbackTesting Sprint alignmentsDependency managementLab support scheduling & deployment phasing 3’rd party integrations
Copyright © 2011, RGCG, LLC 37
Testing-in-Scrum Typical Challenges – Resource Balancing
Resource BalancingBetween traditional, development focused Sprints…Including people, equipment, and simply focus
Falling BehindIf you skew too heavily towards the traditional, testing stabilization Sprints, you’ll lose connection to the next iteration
Falling ForwardIf you skew too heavily towards the development Sprints, you’ll lose the more formal testing & stabilization battle
More often – teams fall behind when they should be falling forward…
Watch out for Waterfall Testing in an Agile World
Copyright © 2011, RGCG, LLC 38
Scrum-in-Testing Typical Challenges – Skill-sets
Agile skill-sets and collaborative expectations are WAY different!
DoDefine agile testing behavior guideline (role & responsibilities)Encourage pairing and strong collaborationAssess your technical capabilities and begin to aggressively fill in any gaps:
Technical domain understanding and direct programming experienceOpen source automation tools experienceCustomer collaborative and UAT experience
Establish guidelines for balancing between Agility & corporate quality and governance expectations
Don’t underestimate the impact that Agility will have towards your traditional teams’ capabilities, approaches and capacity for change
Copyright © 2011, RGCG, LLC 39
Scrum-in-Testing Typical Challenges – Quality Influence
Working with the Product Owner (Customer)Becoming a collaborative partner; defining & automating acceptance testsActively representing the VOC
It’s important to setup clear & broad release criteria for each Sprint and the overall Release
Feature goals; usability goals; acceptance confirmationQuality criteria (defects, coverage, types of testing)Process artifact goals (for example: SOX or other compliance, traceability)
Establishing release readiness (features, quality, interoperability) for PO during Sprint Review (Pass/Fail – goals met?)
Copyright © 2011, RGCG, LLC 40
Scrum-in-Testing Typical Challenges – Metrics & Visibility
As the number of Scrums grow with application size, feedback is gathered more so at the SoS level via –
Cross Sprint burndowns and feature trackingFeedback from the testing team in integration issuesProduct Owner Sprint Review(s) experienceProduct Roadmap progress – across all of the relevant Sprints
The Meta ScrumMasters & Meta Product Owners are actively engaged in defining goals and measuring progress against them
Test teams can / should provide some traditional metrics focusedtowards –
Defects, test coverage & traceability, workflow defect patterns, Sprint release acceptance / goal attainment levels
Copyright © 2011, RGCG, LLC 41
Testing-in-Scrum Typical Challenges – Defects
In pure Agile teams (small teams & projects, quality & testing focused, automation centered) there is little need for traditional defect tracking techniques
They test first by developing unit tests and continuously integrating changes;Establish automated acceptance tests and fixing bugs as they’re found
However, in evolving or large-scale Agile teamsThere is a need for defect coordination across the various product(s) and team(s); Traditional triage, and targeting repairs to specific teams & iterations Product Owner(s) and ScrumMaster(s) are involved in this coordination and delegationTesting teams are at the center of these efforts; guiding the teams towards their Sprint & Product Release goals
Copyright © 2011, RGCG, LLC 42
Wrap-up
Traditional Scrum (and other Agile methods) struggle to operate in:Non-green fieldLegacy based or compliance focusedEnterprise level or large-scale team
environments. This creeps into testing activity as well…
Testing in Scrum in these contexts should include:Strong partnership with the POCollaborative development of a strategy that gradually moves from traditional to agile testingDevelopment of a skewed testing model that support PO / Business and Organizational / Quality needsAwareness of the cultural and skill-set changes that need to be madePatience and emergent (Self-directed) change!
Copyright © 2011, RGCG, LLC 43
Questions?
Thank you!
Copyright © 2011, RGCG, LLC 44
Core Agile ReferencesAugustine, Sanjiv, “Managing Agile Projects”, Addison Wesley, (2005)Beck, Kent, “Extreme Programming Explained – Embrace Change”, Addison Wesley, (2005)Beck, Kent and Fowler, Martin, “Planning Extreme Programming”, Addison Wesley, (2001)Cockburn, Alistair, “Agile Software Development – The Cooperative Game, 2’nd Edition”, Addison Wesley, (2006)Cockburn, Alistair, “Crystal Clear – A Human-Powered Methodology for Small Teams”, Addison Wesley, (2005)Cohn, Mike, “User Stories Applied – For Agile Software Development”, Addison Wesley, (2004)Cohn, Mike, “Agile Estimating & Planning”, Addison Wesley, (2006)Derby, Esther and Larsen, Diane, “Agile Retrospectives – Making Good Teams Great”, Pragmatic Bookshelf, (2006)Highsmith, Jim, “Agile Project Management”, Addison Wesley, (2004)Larman, Craig, “Agile & Iterative Development – A Manager’s Guide”, Addison Wesley, (2004)Leffingwell, Dean, “Scaling Software Agility – Best Practices for Large Enterprises”, Addison Wesley, (2007)Manns, Mary Lynn and Rising, Linda, Fearless Change – Patterns for Introducing New Ideas”, Addison Wesley, (2004)
Copyright © 2011, RGCG, LLC 45
Core Agile ReferencesPoppendieck, Mary & Tom, “Lean Software Development – An Agile Toolkit”, Addison Wesley, (2003)Poppendieck, Mary & Tom, “Implementing Lean Software Development – From Concept to Cash”, Addison Wesley, (2006)Schwaber, Ken and Beedle, Mike, “Agile Software Development with Scrum”, Prentice Hall Publishing, (2002)Schwaber, Ken, “Agile Project Management with Scrum”, Microsoft Press, (2004)Schwaber, Ken, “The Enterprise and Scrum”, Microsoft Press, (2007)Tabaka, Jean, “Collaboration Explained – Facilitation Skills for Software Project Leaders”, Addison Wesley, (2006)
Copyright © 2011, RGCG, LLC 46
Core Agile Web ReferencesMike Cohn – Scrum of Scrums page/picture: http://www.mountaingoatsoftware.com/scrum/scrumteam.php
Jeff Sutherland: http://jeffsutherland.comAgile 2005 paper on Type A, B, C Scrums: http://www.agile2005.org/RP10.pdfAgile 2005 presentation on Scrum II: http://jeffsutherland.com/scrum/ScrumIIAgile2005.pdf#search=%22scrum%20of%20scrums%22
Jeff Sutherland Agile 2006 presentation on Distributed Scrum Teams: www.scrumalliance.org/index.php/content/download/4836/49524/file/SutherlandDistributedScrumAgile2006_v4_28_Feb_2006.pdf
Annual Scrum Gatherings and Agile conferences – www.agile2008.orgAlternate perspective on XP - http://www.softwarereality.com/ExtremeProgramming.jspFirms using Scrum - http://scrumalliance.pbwiki.com/Firms-Using-Scrum?doneSave=1Planning Poker cards - http://contrado.com.au/PP_Cards.pdf
The history of the term SCRUM dates back to a 1986 article by Takeuchi and Nonaka in Harvard Business Review. In it they connected the Rugby term to an iterative model for product development. It's worth a read to simply see the history and connection back to Lean Manufacturing practices - http://apln-richmond.pbwiki.com/f/New%20New%20Prod%20Devel%20Game.pdf
Copyright © 2011, RGCG, LLC 47
Contact Info
Bob GalenRGalen Consulting Group, L.L.C.
Agile focused training, coaching & consulting
PO Box 865, Cary, NC 27512919-272-0719
Scrum Product Ownership – Balancing Value From the Inside Out published by RGCG in 2009.
Software Endgames: Eliminating Defects, Controlling Change, and the Countdown to On-Time Delivery
published by Dorset House in 2005.
Go to www.rgalen.com for purchasing / for order info, misc. related presentations, and papers.