View
112
Download
2
Category
Tags:
Preview:
DESCRIPTION
Are you embarking on a large-scale, globally distributed, multi-team scrum project? Have you already identified the potential testing challenges that lie ahead? Or have you belatedly encountered them and are now working on them in real-time? Five years and more than 200 projects into its agile journey, Dell Enterprise Solutions (ESG) has empirically determined that once a project extends beyond three scrum teams, interesting testing challenges arise—inconsistent “done” criteria, integration testing underscored by epic/story interdependencies across teams, test automation inconsistency, and uncoordinated regression testing. Worse yet, the more teams involved, the less likely it is that a single scrum team has the visibility to validate the overall product from a customer usage perspective as the product evolves through sprints. Geoff Meyer serves up some lessons learned from within the Dell ESG Validation organization as it evolved its agile testing and automation strategies from a waterfall-based environment to one that fully embraced agile Scrum across its entire software product portfolio.
Citation preview
W16
Agile Testing
5/7/2014 3:00:00 PM
Meet Big Agile: Testing on
Large-Scale Projects
Presented by:
Geoff Meyer
Dell, Inc.
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073
888-268-8770 ∙ 904-278-0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com
Geoff Meyer
Dell, Inc.
A test architect in the Dell Enterprise Solutions Group, Geoff Meyer has more than twenty-eight years of experience as a software developer, manager, business analyst and test architect. Geoff helps serve the software development practices of more than 800 development, test, and UX engineers across four Global Design Centers. Geoff is an active member in and contributor to the Agile Austin community.
4/26/2014
1
Agile Testing @ LargeGeoff Meyer, geoffrey_meyer@dell.com
May 2014Last updated: 25 February, 2014
Courtesy: Gulliver's Travels (2010)
Session Objectives
• Challenges to expect with Agile practices in
large-scale, multi-team projects
• Brief overview of Industry Models
• Real-world examples − from a large organization
perspective
• Organizational strategies & considerations
• Test automation considerations
• An In-depth Tools Discussion
• Focused on how to develop Test Automation
IS
IS
NOT
4/26/2014
2
Geoff Meyer
• Dell Inc, 1998 – present
– Test Architect
› Agile Test & Automation Strategy
– Agile Steering Committee Co-Chair
– Global Projects
› 15 Scrum teams (2 regions)
› 11 Scrum teams (4 regions)
› 9 scrum teams (2 regions)
› 7 Scrum teams (4 regions)
• NCR Corp. 1984 – 1998
– SW developer, Project Lead, SW Manager
• B.S. Computer Science, San Diego State University
• Masters Engineering Management - NTU
3
4
Agenda
• The Dell Context and
• Early Agile Testing Challenges
• In Search of… Solutions for Scale
• Adaptations for Testing at Scale
• Automating in an Agile world
• Lessons of Large
Agile @
4/26/2014
3
5
The Context at Dell ESG
http://amazngwallpapers.blogspot.com
Dell Enterprise Solutions GroupGlobal Design and Development using Agile Scrum
6
Noida Design
Center
Bangalore
Design Center
Austin
Design Center
Silicon Valley
Design Center
4/26/2014
4
Dell Enterprise SolutionsSoftware Products
• Server Systems Management
• Converged Infrastructure Systems Management
• Private Cloud Systems Management
• Console Plug-ins (i.e. for SCCM, vCenter…)
7
Common Product Characteristics:
• Embedded Software
• Large hardware support test matrix
• Software is installed in the Data Center
• Enterprise update cycles ~6 months
• Products must function even as underlying HW, FW,
BIOS, and Drivers are upgraded - SUSTAINING
Storage
Network
Compute
The Early Challenges
8
?
http://cleaneatingchelsey.com/
4/26/2014
5
Culture Transition
• Early commitment on Requirements
• Insufficient Detail in Requirements
• Development vs. Test
• A different “School” of Test1
• Fail Early, Inspect & Adapt
91 – Scott Barber “Approaches to Software Testing: An Introduction “
“ “Everybody, every team, every platform, every division, every component is there not for individual competitive profit or recognition, but for contribution to the system as a whole”
W. Edwards Deming
Signs of Trouble• Thursday’s Build
• Planning more stories into sprint than team’s velocity
• High % of carry-overs
• Inability to Automate Acceptance Tests within Sprint
• “Engineering” Stories
Large-Scale
• Cross-team interdependencies block sprint progress
• Large HW configuration validation consumes test resources
• No single team had visibility to the entire product
• Insufficient Product Owner bandwidth
http://www.huffingtonpost.com/2012/03/21/truck-stuck-thomas-
circle_n_1369545.html?p=truck+and+underpass&vm=r
4/26/2014
6
Test Automation Pitfalls
• Minimal guidance to Test beyond “Go forth and automate”
• Black Box Test Mentality
• Automation was interpreted by many as “Automate the UI”
11
• Insufficient SW engineering skills across Test teams
• Architecture(s) not optimized for Automatability
12
In Search of…
Solutions For Scale
4/26/2014
7
Leverage the Community
13
Books
Conferences
Professional Groups
ConsultantsDallas Agile Leaders’ roundtable
Scaled Agile Framework™ Big Picture
© 2008- 2013 Scaled Agile, Inc and Leffingwell, All rights reserved
4/26/2014
8
Agile Testing Quadrants
15
4 - http://lisacrispin.com/wordpress/2011/11/08/using-the-agile-testing-quadrants/
4
ALM Tooling
16
• Multi-team, geographically distributed projects create a necessity for ALM Tooling
–Requirements Management
–User Story Management
–Test Case Management
–Defect Management
–Test Script Repository
4/26/2014
9
Automation Tooling
17
• IDE
• UI Tools
• Scripting Languages
• Services
• Frameworks
18
Agile @ DellAdaptations
4/26/2014
10
Agile Testing @ PG Enterprise Solutions Group How did we go about it?
19
Process/Project
Culture/Organization
X-Large
project (15)
2011 2014
UI Automation
“Automate-First”
Service-level
automation ALM using
TFS
In-sprint
Automation
Accelerate
Automation
Skillset
Maximize
UtilizationAutomation
Architects
Automated
Setup & Config
4 Geography
project (11)
2012
TDD &
Automated
UT
Non-Functional
Scale, Longevity
2010 2013
Test into
Scrum
2009
Collaborative
Seating
Rally
Agile @ DellCore Activities
• Pre-Sprint activities:
– Staffing
– Training - Project tools and processes
– CI/Build environment
– Automation Framework and BVT
• Establish Project-wide ‘Done’ criteria
• End-to-end, short duration User Stories
• Test Automation is included User Story acceptance criteria
• Refresher Workshops for new projects
20
4/26/2014
11
Week 1 Week 2
2 week Sprint Lifecycle Example
Manual Regression
Automated
Regression
Automated
Regression
Automated
Regression
Sprint
Planning
Automated
Regression
Automated
Regression
Sprint
Review
Retro
spective
Demo
Accept
User
Story
Demo
Accept
User
Story
Demo
Accept
User
Story
Demo
Accept
User
Story
Demo
Accept
User
Story
Product Owner (or Proxy)
Scrum Master
Development
Test
Tech Pubs
UI
22
In-Sprint: Testing
• Design/Develop Test cases and Review with team
• Develop test automation and maintain in TFS
• Execute test cases (manual and automated)
• Immediate Feedback (IM, email, in-person) with team of
sightings… do not automatically enter defects
• Verify fixes for defects; re-test if needed
• Maintain test case status (in MTM)
• Cross-team coordination including Weekly SoS
Confidential
4/26/2014
12
Agile @ Dell Roles/Responsibilities
23
Product
Owner
Product Owner Proxy1 per Scrum
Scrum Master1 per Scrum
Development4-5 per Scrum
Test3:1
Tech Pubs1 per 3 scrum team
UI1 per Scrum
Scrum teams
• Small teams
• Co-located
• Multi-disciplined
• Feature-based
• Distributed Projects
• Scrum of Scrums
• Teams formed from
functional silos
…
Release Manager
SW Architect
Test Architect
Automation Architect
Cross-team Test roles
24
• Test Architect
– Defines Test Strategy across all Scrum teams
– Establishes non-Functional Test approach
– Influences consistent Application Architectures
• Automation Architect
– Leads Analysis and POC activities for new automation needs
– Ensures automation consistency for BVT and nightly regression suites
– Safeguards against automation tool proliferation
4/26/2014
13
EXAMPLE - Project-wide “Done” criteria
25
Code
– Junit Tests
– SONAR
– Code Reviews completed
– All User Story tasks completed
– Demo and accepted by Product Owner
– All known error conditions caught and handled appropriately
– Code is Globalized
Test
– Acceptance Tests Reviewed with scrum team
– Acceptance Tests documented in Tool
– All planned Acceptance Tests for User Story has been attempted
– Acceptance Tests executed against Test system configuration
– Defects that are “must fix” before releasing must be fixed prior to story acceptance
– Service-level automated tests
– Service-level automated Tests are globalized
Large-Scale Complexities
26
Characteristic Description
Large-Scale Project consists of more
than 4 Scrum teams
Interdependency Requirements
implemented across
multiple Scrum teams
Extensive
Configuration
Matrix
Extensive HW or SW
configurations
Waterfall
Intersection
One or more
components are
managed using Waterfall
Geography Project members are
Geographically dispersed
4/26/2014
14
Development
Usability
Test
Wireframes
Architecture
Automation FW/Tools
Release Planning
Pre-sprints
Pre-sprints
HW resources
HW resources
Pre-sprints
: Agile with Adaptations
27
Hardening
1 2 3 N-1 N
1 2 3 N-1 N
1 2 3 N-1 N
… Sprints
Define Plan Develop Launch
OLP
…
Software System Test
Extended Sprint Test
Stability
Agile @
Extended Sprint TestExtended Hardware Configurations
• Optimizes HW utilization and cost in support of Scrum teams
• Owns Test Execution against extendedconfigurations
• Scrum team:
– Establishes Acceptance criteria
– Identifies all Test scenarios
– Owns Test Execution against reference configuration(s)
– Develops in-sprint automation tests
28
Configurations
Positive X X X X
Negative X X X
Boundary X X X
Stress X X
Scalability X X X
Concurrency X X X
Globalization X X
Candidate for
Extended Sprint Test
4/26/2014
15
Pre-SST
Software System Test“Alpha Customer”
29
Hardening
• Workflow Design
• Test Analysis
• Test Design
• Test Development
• Test Automation
Development
• Test Execution
• Regression Test
• Test Execution
• Regression Test
SST
IS IS NOT
Requirements-based Functional testing
Customer-usage based First time integration test of workflows
Interoperability coverage of HCL in
specific customer environments
Exhaustive interoperability matrix
coverage of the HCL
Three phased approach Performance baseline testing
SAFe1 &
30
Agile @
1- Courtesy of Dean Leffingwell and Al Goerner of Valtech
Scrum teams
System team
4/26/2014
16
PG Enterprise Validation
Agile & Waterfall Converged Model
Dell Confidential31
SST
Hardware Product Test
EST
FinalEST NEST 1
Sprint1 Sprint2Sprint N-1
�
SprintN
Sprint1 Sprint2Sprint N-1
SprintN
Sprint1 Sprint2Sprint N-1
SprintN
Hardening
Stability
Sprint
Extended Sprints
SW Regression
Test
Solution stack with
software focus
�
SST
EST Exit
CCB Review
Code Freeze
A-Rev
Final
Extended Regression
F
V
PT
Stability
SST
A-Rev
Candidate
SSI
�Stability
SprintRC-0
Regression
Regression
HW-Specific User Stories Complete
HW Enablement Features Complete
Vendor Deliverables Required
100% PT Attempted
All Stories Accepted
Feature Complete
X-on-A Stories
Accepted
PG Enterprise Validation
Agile @ DellGeography
32
• Co-location of a Scrum team is always preferred
• Limit geographical distribution of Scrum team to no more than two time-zones
• Scrum teams within a project can be distributed
4/26/2014
17
33
Automating in an Agile World
Opportunities for Automation
• It’s not just about automating test cases
• Can also provide efficiencies to:
–Test Preparation, Setup and Configuration
–Non-Functional Testing (“ility”)
› Longevity, Scale and Performance Characterization
–Compatibility Testing (Solution and Device)
34
4/26/2014
18
Test Preparation
• Bare-metal Deployment
• Setup & Configuration
– OS Provisioning and Configuration
• Test case staging
• Environment Cleanup/Baseline
• Virtualization-based test environments
35
Test Content Development
• UI automation on Customer Usage workflows
36
CAUTION: Application Architecture can be an enabler or inhibitor
• Unit Test Automation
• In-Sprint, Automated Acceptance Tests
• Web-services (or CLI) Test Automation for Functional Testing
• Automated CI (UT, Build, BVT)
2 - Agile Testing: A Practical Guide for Testers and Agile teams
2
4/26/2014
19
Application Architecture Matters
• Does it facilitate testing at the API/Services level?
• Does the Business/Error Logic reside below the services level?
• Does the UI architecture support test automation other than record-playback?
37
Large Scale Organizations:• Standardize UI architectures across the product
portfolio• HTML5, Flex, Silverlight…
• Standardize Service architectures across the
product portfolio• SOAP, REST, CLI, API
Non-Functional Testing
• Performance Characterization
• Longevity
• Stress
• Scale
• Concurrency
38
Often times analysis of these areas are simply too cost-prohibitive to be done manually
3- http://lisacrispin.com/wordpress/2011/11/08/using-the-agile-testing-quadrants/
3
4/26/2014
20
Confidential39
Functional
Validation
Performance-
Characterization and
non-Functional Validation
Longevity
11 M2 1.0
SST
Baseline
M1
Scale
Concurrency
Simulation
L
a
r
g
e
-
S
c
a
l
e
L
a
r
g
e
-
S
c
a
l
e
12 13 17 18 19�
Sprint 15
Performance Characterization Timeline
14 16
Sprint 21
Key Dependency:
• 3rd party HW available
• 3rd party user stories
completed
The Lessons of ‘Large’
40
Courtesy: Gulliver's Travels (2010)
4/26/2014
21
The Lessons of ‘Large’
1. Co-located Teams
2. Project-wide ‘Done’ Criteria
3. Culture of Automation
4. System Test Team
5. Test Architect
6. ALM Tooling
Questions?
42
4/26/2014
22
Resources• Agile Manifesto - http://agilemanifesto.org/
• Articles/Blogs:
– http://support.smartbear.com/articles/testcomplete/automated-testing-agile-environment/ -SmartBear
– http://lisacrispin.com/wordpress/2011/11/08/using-the-agile-testing-quadrants/ - Lisa Crispin
– http://www.stickyminds.com/sitewide.asp?Function=edetail&ObjectType=COL&ObjectId=17793&tth=DYN&tt=siteemail&iDyn=2 – Rajini Padmanaban
– Beyond Agile Testing: http://www.utest.com/int-v1/a/beyond-agile-testing - uTest
– Testing and Checking Refined: http://www.satisfice.com/blog/archives/856 - James Bach and Michael Bolton
• Books:
– Scaling Software Agility: Best Practices for Large Enterprises – Dean Leffingwell
– How Google Tests Software – James Whittaker, Jason Arbon, Jeff Carollo
– Agile Testing: A Practical Guide for Testers and Agile teams – Lisa Crispin, Janet Gregory
– http://www.dorothygraham.co.uk/automationExperiences/index.html - Dorothy Graham
• Presentations:
– Approaches to Software Testing: An Introduction – Scott Barber
– Scaled Agile Framework Big Picture – Dean Leffingwell43
Backup
44
Recommended