Embed Size (px)
Citation preview
Agile Test Strategies in Practice:Tales of Large-scale Agile projects
8 May 2015
Geoff Meyer, [email protected]
SW Engineering Director, Dell Inc
• Introduction to Key Agile Testing concepts
• Real-world examples − from multiple, large-scale projects
• Organizational strategies & considerations
• Test automation strategies and tooling considerations
• Test Data Management
• An In-depth Tools Discussion
• Focused on how to develop Test Automation
IS
IS
NOT
Session Objectives
Geoff Meyer, SW DirectorDell Commerce Services Group
2015
2010
2005
2000
1998
1995
1990
1984
Personal Family is paramount; my wife is a Professional Networking queen and my kids make me proud!
LivesAustin, TX
LivedSan Diego, Twin Cities
HobbiesTravel, Avid fan of NFL, NHL, EPL, MLB, Golfer, Monday Morning Quarterback
Business SpecialtyChange Agent and champion for Agile transformations
PassionSolving problems with great people and enjoying every workday
Operating ModelEmbrace Change and Continuous Learning
RolesDirector, Architect, Program Manager, Manager, Developer
My Role @ Dell Leads Test and DevOps functional areas for Dell.com
MissionImplement Continuous Testing, Integration and Delivery practices
GoalTransform Dell.com into a SW Development engine characterized by Quality, Predictability and Continuous Delivery
Agenda
• Agile Testing Concepts– Organization and
Architecture
– Agile Testing
– Test Automation
– Continuous Testing
– Large-Scale Agile and SAFE
• Agile Projects in Action
Agile Roles/Responsibilities
5
Program Product Owner
Product Owner
Scrum Master
Development
Test
• Small teams
• Co-located
• Multi-disciplined
• Feature-based…
Program ManagerSW ArchitectTest Architect
DevOps
• Distributed Projects
• Scrum of Scrums
Week 1 Week 2
Agile Sprint Testing Cycle
Manual Regression
AutomatedRegression
AutomatedRegression
AutomatedRegression
Sprint Planning
AutomatedRegression
AutomatedRegression
Sprint Review
Retrospective
Demo
Accept
User Stor
yDemo
Accept
User Stor
y
Demo
Accept
User Stor
y
Demo
Accept
User Story
Demo
Accept
User Story
Product Owner Scrum
MasterDevelopment
Test
Automation Pyramid
7CAUTION: Architecture (and Organization!) can be an enabler or inhibitor
“Automate Within the Sprint. Automation on a Scrum project is not optional.” - Mike Cohn
3Agile Testing Quadrant
8More Agile Testing: Learning Journeys for the Whole Team. Gregory, Janet; Crispin, Lisa (2014-09-30).
Feature Team
System Team(s)
UATE2E
Performance
Automation Tooling
9
• IDE
• UI Tools
• Scripting Languages
• Services
• Unit
• Frameworks
MS-Test
Scaled Agile Framework™ Big Picture
© 2008- 2013 Scaled Agile, Inc and Leffingwell, All rights reserved
Agile Projects in Action @ Scale
Project ProfilesApplication Architecture
Presentation Layer
Business/Service Layer
Data Access Layer
MoeSystems Mgmt
Application
CurleyeCommerce UI
Application
LarryeCommerce
Services
OrganizingTeam Roles/Responsibilities
13
Program Product Owner
Product Owner
Scrum Master
Development
Test
Agile Test Strategy: Moe
All Agile Testing managed within the program
Agile Test Strategy: Curley
UAT and ‘ility’ testing conducted by external teams
UAT Regression tests automated by the program team
Agile Test Strategy: Larry
‘ility’ testing conducted by external teams
Automation Strategy: Moe
UI Automation limited to Scenario-based workflows
In-Sprint Automation at the Unit and Service Levels
Confidential18
FunctionalValidation
Performance-Characterization and non-Functional Validation
Longevity
11 M2
1.0
SST
Baseline
M1
Scale
Concurrency
Simulation
Large-Scale
Large-Scale
12 13 17 18 19…
Sprint 15
‘ility Testing’; Moe
14 16
Sprint 21Key Dependency:• 3rd party HW available• 3rd party user stories
completed
Automation Strategy: Curley
UI Automation to Scenario-based workflows+UI functional
tests
Functional Test Automation done In-Sprint at the Unit and UI Levels
Services Virtualization to support Continuous Testing
Automation Strategy: Larry
Functional Test Automation done in-Sprint at the Unit and Service Levels
MS-Test
Services Virtualization to support Continuous Testing
Continuous Integration & Automated Testing: Moe
21
Automated TestingContinuous BVT
Team #1 Feature Test• Exploratory Testing• Acceptance Testing
Team #2 Feature Test• Exploratory Testing• Acceptance Testing
Team #N Feature Test• Exploratory Testing• Acceptance Testing
CI/Build
TFS TeamBuild
To Factory
Continuous Integration & Continuous Testing: Curley
22
Continuous TestingContinuous BVT
SIT
App Test VMTeam #1 Feature Test• Exploratory Testing• Acceptance Testing
App Test VMTeam #2 Feature Test• Exploratory Testing• Acceptance Testing
App Test VMTeam #N Feature Test• Exploratory Testing• Acceptance Testing
…
CI/Build
TFS TeamBuild
To Staging & Production
Continuous Delivery
23
CI/Build
BVT Tests
BVT AutomatedSmoke
DVTUAT
Production
BaselineLoad
Stress
Performance
E2E TestsUAT regression
Staging
UATFunc. regressionUAT regression
E2E
SIT
Unit testIntegration test
Dev Test
Unit testIntegration test
Dev Test
Unit testIntegration test
Dev Test
Unit testIntegration test
Dev TestAcceptance TestsExploratory Tests
UI TestsFunc. RegressionUAT regression
App Test
Acceptance TestsExploratory Tests
UI TestsFunc. RegressionUAT regression
App Test
Acceptance TestsExploratory Tests
UI TestsFunc. RegressionUAT regression
App Test
Unit testIntegration test
Dev Test
Static AnalysisAuto. Unit TestGated Check-ins
Minutes Minutes Accept
Auto Depoy
Auto Deploy
Zero-TouchDeploy
Auto Deploy
• Automated Smoke, DVT, BVT & Regression Suites• Continuous Integration–Fast feedback/Green
Builds• On-demand Environment Provisioning &
Deployment• Local Control of Environments: Mocks & Data
The Lessons of ‘Large’
• There is no cookie-cutter Test Strategy
• Strive for co-located Teams
• Establish Culture of Automation
• Create Cross-team roles
• Inspect and Adapt
Dell - Restricted - Confidential
Recommended Readings
25
Questions?
