Large-Scale Agile Test Automation Strategies in Practice

AT8 ConcurrentSession11/12/1511:30am

“Large-Scale Agile Test Automation Strategies in Practice”

Presented by:

Geoff Meyer

Dell, Inc.

Broughttoyouby:

340CorporateWay,Suite300,OrangePark,FL32073888-268-8770·904-278-0524·[email protected]·www.techwell.com

Geoff Meyer Dell, Inc.

As director of SW Test and DevOps for Dell.com, Geoff Meyer leads the definition and implementation of the Agile Scrum Software Test processes, test automation strategy, and the adoption of continuous delivery practices for Dell’s eCommerce site. A seventeen-year veteran of Dell, Geoff’s roles have spanned the software development spectrum including test architect, program manager, software manager, and developer. Geoff has been an active member of the Agile Austin community since 2010.

Agile Test Strategies in Practice:Tales of Large-scale Agile projects

12 November 2015

Geoff Meyer, [email protected]

SW Engineering Director, Dell Inc

• Introduction to Key Agile Testing concepts

• Real-world examples − from multiple, large-scale projects

• Organizational strategies & considerations

• Test automation strategies and tooling considerations

• Test Data Management

• An In-depth Tools Discussion

• Focused on how to develop Test Automation

IS

IS NOT

Session Objectives

Geoff Meyer, SW DirectorDell Commerce Services Group

2015

2010

2005

2000

1998

1995

1990

1984

Personal Family is paramount; my wife is a Professional Networking queen and my kids make me proud!

LivesAustin, TX

LivedSan Diego, Twin Cities

HobbiesTravel, Avid fan of NFL, NHL, EPL, MLB, Golfer, Monday Morning Quarterback

Business SpecialtyChange Agent and champion for Agile transformations

PassionSolving problems with great people and enjoying every workday

Operating ModelEmbrace Change and Continuous Learning

RolesDirector, Architect, Program Manager, Manager, Developer

My Role @ Dell Leads Test and DevOps functional areas for Dell.com

MissionImplement Continuous Testing, Integration and Delivery practices

GoalTransform Dell.com into a SW Development engine characterized by Quality, Predictability and Continuous Delivery

Agenda

• Agile Testing Concepts

– Organization and Architecture

– Agile Testing

– Test Automation

– Continuous Testing

– Large-Scale Agile and SAFE

• Agile Projects in Action

Agile Roles/Responsibilities

5

Program Product Owner

Product Owner

Scrum Master

Development

Test

• Small teams

• Co-located

• Multi-disciplined

• Feature-based…

Program ManagerSW ArchitectTest Architect

DevOps

• Distributed Projects

• Scrum of Scrums

Week 1 Week 2

Agile Sprint Testing Cycle

Manual Regression

AutomatedRegression

AutomatedRegression

AutomatedRegression

Sprint Planning

AutomatedRegression

AutomatedRegression

Sprint Review

Retrospective

Demo

Accept

User Story

Demo

Accept

User Story

Demo

Accept

User Story

Demo

Accept

User Story

Demo

Accept

User Story

Product Owner Scrum MasterDevelopment

Test

Automation Pyramid

7CAUTION: Architecture (and Organization!) can be an enabler or inhibitor

“Automate Within the Sprint. Automation on a Scrum project is not optional.” - Mike Cohn

3Agile Testing Quadrant

8More Agile Testing: Learning Journeys for the Whole Team. Gregory, Janet; Crispin, Lisa (2014-09-30).

Feature Team

System Team(s)

UATE2E

Performance

Automation Tooling

9

• IDE

• UI Tools

• Scripting Languages

• Services

• Unit

• Frameworks

MS-Test

Scaled Agile Framework™ Big Picture

© 2008- 2013 Scaled Agile, Inc and Leffingwell, All rights reserved

Agile Projects in Action @ Scale

Project ProfilesApplication Architecture

Presentation Layer

Business/Service Layer

Data Access Layer

MoeSystems Mgmt

Application

CurleyeCommerce UI

Application

LarryeCommerce

Services

OrganizingTeam Roles/Responsibilities

13

Program Product Owner

Product Owner

Scrum Master

Development

Test

Agile Test Strategy: Moe

All Agile Testing managed within the program

Agile Test Strategy: Curley

UAT and ‘ility’ testing conducted by external teams

UAT Regression tests automated by the program team

Agile Test Strategy: Larry

‘ility’ testing conducted by external teams

Automation Strategy: Moe

UI Automation limited to Scenario-based workflows

In-Sprint Automation at the Unit and Service Levels

Confidential18

FunctionalValidation

Performance-Characterization and non-Functional Validation

Longevity

11 M2 1.0

SST

Baseline

M1

Scale

Concurrency

Simulation

Large-Scale

Large-Scale

12 13 17 18 19…

Sprint 15

‘ility Testing’; Moe

14 16

Sprint 21Key Dependency:• 3rd party HW available• 3rd party user stories

completed

Automation Strategy: Curley

UI Automation to Scenario-based workflows+UI functional

tests

Functional Test Automation done In-Sprint at the Unit and UI Levels

Services Virtualization to support Continuous Testing

Automation Strategy: Larry

Functional Test Automation done in-Sprint at the Unit and Service Levels

MS-Test

Services Virtualization to support Continuous Testing

Continuous Integration & Automated Testing: Moe

21

Automated TestingContinuous BVT

Team #1 Feature Test• Exploratory Testing• Acceptance Testing

Team #2 Feature Test• Exploratory Testing• Acceptance Testing

Team #N Feature Test• Exploratory Testing• Acceptance Testing

CI/Build

TFS TeamBuild

To Factory

Continuous Integration & Continuous Testing: Curley

22

Continuous TestingContinuous BVT

SIT

App Test VMTeam #1 Feature Test

• Exploratory Testing• Acceptance Testing

App Test VMTeam #2 Feature Test


App Test VMTeam #N Feature Test


…

CI/Build

TFS TeamBuild

To Staging & Production

Continuous Delivery

23

CI/Build

BVTTests

BVT AutomatedSmoke

DVTUAT

Production

BaselineLoadStress

Performance

E2E TestsUAT regression

Staging

UATFunc. regressionUAT regression

E2E

SIT

Unit testIntegration test

Dev Test


Dev Test


Dev Test


Dev TestAcceptance TestsExploratory Tests

UI TestsFunc. RegressionUAT regression

App Test

Acceptance TestsExploratory Tests


App Test

Acceptance TestsExploratory Tests


App Test


Dev Test

Static AnalysisAuto. Unit TestGated Check-ins

Minutes Minutes Accept

Auto Depoy

Auto Deploy

Zero-TouchDeploy

Auto Deploy

• Automated Smoke, DVT, BVT & Regression Suites• Continuous Integration–Fast feedback/Green Builds• On-demand Environment Provisioning & Deployment• Local Control of Environments: Mocks & Data

The Lessons of ‘Large’

• There is no cookie-cutter Test Strategy

• Strive for co-located Teams

• Establish Culture of Automation

• Create Cross-team roles

• Inspect and Adapt

Dell - Restricted - Confidential

Recommended Readings

25

Questions?