Automation Culture:Essential to Agile SuccessAgile Austin

Geoff Meyer, geoffrey_meyer@dell.com

October 2013Last updated: September 29, 2013

Session Objectives• Challenges to expect when adopting test

automation during your Agile transition

• Practical solutions to what & when to automate

• Real-world examples − from a large organization perspective

• Challenges automating in a HW-dependent environment

• For organizational leaders driving test strategy

• A Tools Discussion

• Focused on how to develop Test Automation

IS

IS

NOT

Introductions

3

4

Geoff Meyer• Dell Inc, 1998 – present

– Test Architect

› Agile Test & Automation Strategy

– Agile Steering Committee Co-Chair

– Global Projects

› 15 Scrum teams (2 regions)

› 11 Scrum teams (4 regions)

› 9 scrum teams (2 regions)

› 7 Scrum teams (4 regions)

• NCR Corp. 1984 – 1998

– SW developer, Project Lead, SW Manager

• B.S. Computer Science, San Diego State University

• Masters Engineering Management - NTU

5

Agenda

• Why Automation is essential to Agile

• The Dell Context and Agile @ Dell

• Challenges and Common Pitfalls

• The Automation Landscape

• Foundations of an Automation Culture

• Care and Feeding of the Automation Culture

The Need,

The Challenge,

The Pitfalls

6

Ø

Why is Automation So Important in Agile?

• Near-term

– Ensures that you don’t break what you just built

– Provides safety net for developers & rapid feedback to new changes

– Continuous Integration and use of Build Verification Test (BVT)• Long-term

– Maximizes velocity of Scrum team (increased value creation)

– Creates capacity for Exploratory and ad-hoc Testing

– Enables activities that can’t be done cost-effectively by humans

And if you don’t…Quality is at risk from an unmanageable regression suite

Differences with Automation in Waterfall?

In Waterfall…

• Automated tests are derived from the backlog of completed features– In Agile, Automation can be incorporated in the

requirement

• Testing and automation is performed after Development is “complete”

• Focus is on first-time discovery of defects and optimizing your test coverage

vs. Agile… where automation development is ongoing and provides immediate feedback throughout development

8

9

The Context @ Dell

http://amazngwallpapers.blogspot.com

“Apply Test Automation in the context of your organizational automation needs”

--- Bob Galen, Agile Coach

10

Dell Enterprise Solutions GroupGlobal Design and Development using Agile Scrum

Noida Design Center

Bangalore Design Center

Austin Design Center

Silicon Valley Design Center

11

Dell Enterprise SolutionsSoftware Products

• Server Systems Management

• Converged Infrastructure Systems Management

• Private Cloud Systems Management

• Console Plug-ins (i.e. for SCCM, vCenter…)

Common Product Characteristics:

• Large hardware support test matrix

• Software is installed in the Data Center

• Enterprise update cycles ~6 months

• Products must function even as underlying HW, FW, BIOS, and Drivers are upgraded - SUSTAINING

Agile @ Dell Roles/Responsibilities

12

Product Owner

Product Owner Proxy~1 per Scrum

Scrum Master1 per Scrum

Development4-5 per Scrum

Test3:1

Tech Pubs1 per 3 scrum teams

UI1 per Scrum

Scrum teams

• Small teams

• Co-located

• Distributed Projects

• Teams formed from functional silos

13

Agile @ Dell with Adaptations

Hardening

1 2 3 N-1 N

1 2 3 N-1 N1 2 3 N-1 N

…

Release

Exit

FeatureComplet

e

Code Freeze

Sprints

Release Plan

Define Plan Develop LaunchOLP

…

Software System Test

Extended Sprint Test

Stability

Pitfalls encountered at

• Development didn’t historically automate unit tests

• Build teams were staffed with non-Build practitioners

• Minimal guidance to Test beyond “Go forth and automate”

14

• Black Box Test Mentality

• Architecture(s) not optimized for Automatability

• Automation was interpreted by many as “Automate the UI”

• Insufficient SW engineering background across Test teams

Which Resulted In…

• Over-emphasis on UI automation

• Test Automation not keeping up within the sprint

• Feature Devotion1

• Multiple automation tools & licenses

• Test scripts not designed for re-use

15

1 – “A nasty condition where people start valuing ticking off features more than tracking the real outcome of the project.”--- Martin Fowler

16

The Automation Landscape

http://amazngwallpapers.blogspot.com

Opportunities for Automation

• It’s not just about automating test cases

• Can also provide efficiencies to:

– Test Preparation, Setup and Configuration

– Non-Functional Testing (“ility”)

› Longevity, Scale and Performance Characterization

– Compatibility Testing (Solution and Device)

17

Test Preparation

• Bare-metal Deployment

• Setup & Configuration– OS Provisioning and

Configuration

• Test case staging

• Environment Cleanup/Baseline

• Virtualization-based test environments

18

Test Content DevelopmentWhole team ownership of QA & Automation

• UI automation on Customer Usage workflows

19

CAUTION: Application Architecture can be an enabler or inhibitor

• Unit Test Automation

• In-Sprint, Automated Acceptance Tests

• Web-services (or CLI) Test Automation for Functional Testing

• Automated CI (UT, Build, BVT)

2 - Agile Testing: A Practical Guide for Testers and Agile teams

2

20

Application Architecture Matters

• Does it facilitate testing at the API/Services level?

• Does the Business/Error Logic reside below the services level?

• Does the UI architecture support test automation other than record-playback?

For Large organizations:– Standardize UI architectures across the product portfolio

› HTML5, Flex, Silverlight…

– Standardize Service architectures across the product portfolio› SOAP, REST, CLI, API

Prioritize and Identify what NOT to Automate

What NOT to Automate

• Everything

• GUI – Except for High-value customer usage flows after the UI is stable

• Tests that would only find low severity bugs

– Where the product is unchanging21

Prioritize High ROI

1. BVT Candidate Acceptance Test

– Test of core functionality that executes in a short duration

2. Core Functionality to be run in Nightly Regression

3. Sustaining Test Candidate Test

– Functional test that verifies the application can withstand subsystem changes

Non-Functional Testing

• Performance Characterization

• Longevity

• Stress

• Scale

• Concurrency

22

Often times analysis of these areas are simply too cost-prohibitive to be done manually3- http://lisacrispin.com/wordpress/2011/11/08/using-the-agile-testing-quadrants/

3

23

Foundations of an Automation Culture

Images.yahoo.com

Keys To A Culture Transition

• Development vs. Test

• A different “School” of Test4

• Evolve from Functional Responsibility

…to Whole Product Ownership

244 – Scott Barber “Approaches to Software Testing: An Introduction “

25

Establishing an Automation Culture

• Identify• Near and long-term automation focus areas• Inventory the culture and skillset of organization

• Establish– Tooling and Infrastructure Standardization– Your ‘community’

• Develop– Workforce transition plan

• Organize– Embedded vs. Specialized Automation team– Re-align project staffing

• Operationalize

26

IdentifyFocus Areas for Automation

4 - http://lisacrispin.com/wordpress/2011/11/08/using-the-agile-testing-quadrants/

4

Establish

• Tooling and Infrastructure– Tooling Standards and Frameworks– Team to develop/manage home-grown tools– Lead tool evaluations to meet new architectures

& technologies

• Community: Internal and External

27

Multiple automation Tools/FW’s may be needed to automate at layers of testing:

• Unit level• Middle tier• UI-driven

Confidential28

Develop & Organize

• Adjust Hiring Practices

• Re-align Project staffing

• Embedded Automation vs. Specialized Team

• Training– New-hire

– Ongoing

29

Operationalize

• Whole team commitment in words and actions

• Include Test Automation in Acceptance Criteria

– Unit

– Acceptance

• Continuous Integration and BVT

• Establish Metrics, Governance & Rewards

Automation @ PG Enterprise Solutions Group How did we go about it?

30

Process/Project

Culture/Organization

In-Sprint Automatio

n

Established ALT

2011 2014

UI Automation

“Automate-First”

Service-level automation(CLI & Web-

services)

IVT Networking & Servers

Automated BVT

IVT Device

Accelerate Automation

Skillset

Maximize Utilization

Automation Architects

Setup & Config

Internal Training

Programs

2012

TDD & Automate

d UT

Non-FunctionalScale,

Longevity

2010

Automation Day

2013

Automation Content Libraries

Care and Feeding of the Automation Culture

31

32

Maintaining the Automation Culture

• Operationalize Automation– Include Test Automation in the Acceptance Criteria

• Encourage Community Practice and Participation– External - Industry User Groups & Conferences– Internal - Brown-bag sessions, Showcases, Mini-conference

• For Large organizations:– Overcome skillset deficiencies by adjusting staffing

strategy– Automation Leadership Team, Automation Architect(s)– Continually monitor the alignment of Architecture, Dev and

Test– Develop reward systems

Recognizing a Successful Automation Culture

• Automation is a shared responsibility

• Automatability is a key architectural consideration

• Continuous Integration & BVT

• Teams are staffed to include an automation skillset

• Automation is operationalized

34

Resources• Agile Manifesto - http://agilemanifesto.org/

• Articles/Blogs:– http://support.smartbear.com/articles/testcomplete/automated-testing-agile-environ

ment/ - SmartBear

– http://lisacrispin.com/wordpress/2011/11/08/using-the-agile-testing-quadrants/ - Lisa Crispin

– http://www.stickyminds.com/sitewide.asp?Function=edetail&ObjectType=COL&ObjectId=17793&tth=DYN&tt=siteemail&iDyn=2 – Rajini Padmanaban

– Beyond Agile Testing: http://www.utest.com/int-v1/a/beyond-agile-testing - uTest

– Testing and Checking Refined: http://www.satisfice.com/blog/archives/856 - James Bach and Michael Bolton

• Books:– Scaling Software Agility: Best Practices for Large Enterprises – Dean Leffingwell

– How Google Tests Software – James Whittaker, Jason Arbon, Jeff Carollo

– Agile Testing: A Practical Guide for Testers and Agile teams – Lisa Crispin, Janet Gregory

– http://www.dorothygraham.co.uk/automationExperiences/index.html - Dorothy Graham

• Presentations:– Approaches to Software Testing: An Introduction – Scott Barber

– Agile Testing: Challenges Beyond the Easy Contexts – Bob Galen

Questions?

35