Building a Test Automation Strategy for Success

© Utopia Solutions

Building a Strategy for Test Automation Success

Lee Barnes, CTO Utopia Solutions

© Utopia Solutions

Agenda

• What is the end goal? • Why do we need a strategy? • What do we need to consider? • What should a strategy look like?

Test Automation Strategy Overview

Interactive Case Study

© Utopia Solutions

SECTION 1 What is Our End Goal?

© Utopia Solutions 4

Test Automation Success Criteria

• Issue detection and recovery • Accurate verification • Unattended execution

Reliable

• Minimum sensitivity to application and test case changes

• Test case definition separate from automation code

Maintainable

• Test coverage expanded efficiently • Automated test cases created by non-

automation resources Scalable


Test Automation Approach

Test Automation Framework

Automated Test Cases Test Results

Test Definition

Test Execution

Non-technical / Test Analysts & Subject Matter Experts

Technical / Automation Engineers

© Utopia Solutions

SECTION 2 Why Do We Need a Strategy?


Why Test Automation Fails

Organizational • Lack of support • Unrealistic expectations • Disjointed org structure • Missing skill sets

Process • Immature testing process • Segregated processes • Lack of test environment / data

control

Technical • Poor tool support for AUT • Under or over design automation

approach


Are We Ready?

© Utopia Solutions

SECTION 3 What Do We Need to Know First?


Test Automation Readiness Assessment

Org

• Structure • Support • Goals and

Expectations • Resources and Skills

Process

• Test Maturity • Test Assets • Process Integration • Test Environment /

Data

Tech • AUT Technology • AUT / Tool

Integration

Assessment Topics

© Utopia Solutions

Where Do We Get the Information?

Assessment Input

Stakeholders

Existing Assets

AUTs

© Utopia Solutions

Ideal Organizational Attributes

Structure

• Centralized testing function

• Testing is a peer of development

• Automation roles are defined

Goals and Expectations

• Automation is an investment

• 100% automation is not likely

• Automation requires specialized skills

• Automation will free up resources to work on high-value tasks

Support

• Executive management has visibility to automation

• Automation has a dedicated budget

Resources and Skills

• Resources are dedicated to automation

• Resources are skilled in automation as a process AND technology

© Utopia Solutions

Ideal Process Attributes

Test Process

• Test objectives are clear and measurable

• Test cases are defined and documented

• Expected results are predictable

SDLC Integration

• AUT in a frequent release state with visibility to future roadmap

• Visibility to system changes as early as possible in release cycle

Test Environment

• Test environment dedicated to automation

• Test data controlled

© Utopia Solutions

Ideal Technology Attributes

AUT Technology

• Multiple AUTs built on common technology and/or development framework

• Consistent use of UI objects

• Consistent navigation • Consistent presentation

of user feedback (warnings, errors, etc.)

Tool / AUT Integration

• All UI objects supported by tool

• Any alternative interfaces (web services, API, DB, etc.) supported by tool

• All target end user environments supported by tool

© Utopia Solutions

SECTION 4 Common Assessment Findings


Organizational

Distributed Testing /

Automation Efforts

App App

App

App • Uneven automation

resource demand • Inconsistent approach • Duplicated efforts • Restricted skill growth

and knowledge sharing

Leads to…


Organizational

AUT A

AUT B AUT C

AUT D Centralized

Test Automation Team

Common Resources, Processes & Technology

• Flatten demand curve • Accelerate skills

development • Leverage reusable

knowledge and technology

Benefits

Centralize Automation Activities


Process

Analyze Design Build

Test Deploy

No ability to identify impact of app changes on automated test suite

Unexpected / Unpredictable Maintenance


Process

Analyze Design Build Test Deploy

Capture potential impacts as early as possible to enable proactive maintenance

Capture App Changes

Analyze Impact Maintain Execute

Automation Activities

Application SDLC


Environment

Shared Test Environment

Uncontrolled Test Data

Integration

Functional

Performance

Integration Test

• Schedule conflicts • Unreliable execution • Unnecessary “hardening”

of the test suite

Leads to… Automation


Environment

Shared Test Environment

Controlled Test Data

Integration

Functional

Performance

Integration Test

Dedicated Environment for Automation

Test Data Controls

Dedicated Environment

Controlled Test Data

Automation

• Restore to known state • Partition data related

to specific test activities

• Create required data each execution

© Utopia Solutions

SECTION 5 What Does a Strategy Look Like?


Mobile Test Strategy Topics

Test Automation

Strategy

Process & Organization

Environment

Technical Resources

Scope & Roadmap


Automation Strategy – Process and Organization

• Political and financial support • Realistic expectations • Centralized automation function

Key Organizational Components

• Test objectives and coverage goals • Integration with SDLC → proactive

maintenance

Key Process Components


Automation Strategy – Environment

• Dedicated environment for execution • Controlled test data • Controlled code migration

Key Environment Components


Automation Strategy – Resources

• Roles • Skills • Time

Key People Components

• AUT test environment(s) • Test suite development • Test suite execution

Key Hardware Components

• Test tool • Supporting utilities

Key Software Components


Automation Strategy – Scope

• Test data setup • AUT / environment preparation • Complex / tedious results verification

Test Support Tasks

• Applications built on common technology • Applications with frequent releases & friendly UI’s

Application Selection

• Frequently tested • High test case to UI ratio • Predictable results • Easy to automate

Test Case Selection


Automation Strategy – Roadmap

• Begin with required organizational and process changes • Overlap with base framework and minimal coverage • Enhance and add test coverage over small iterations

Create Implementation Timeline

Phase n Phase 1

Continuous Process & Organizational Improvements

Base Framework

Initial Coverage

AUT 1

Framework Enhancements

Expand Coverage

Initial Coverage

Phase 2

AUT 2

Expand Coverage

Expand Coverage

Initial Coverage

Phase 3

AUT 3

© Utopia Solutions

SECTION 6 Case Study


Key Assessment Findings

Organization • All enterprise testing activities

currently performed centrally • Management believes automation

is a tool implementation

Process • Manual tests are well structured,

but not maintained consistently • Rely on SME’s to execute tests

Environment • Manual regression test performed

in a isolated environment • Test data refreshed in all

environments once per quarter

Technology • All AUTs have browser based UI • Developed on disparate

technologies • Test tool supports all AUT

components

Resources • Test organization consists of non-

technical SME’s • System performance (mobile) • Device performance


Key Summary Points

• DO NOT automate without a clear vision and plan • Understand your readiness from these perspectives

• Organization • Process & Environment • Technical • Resources

• Implement in small phases


Questions…

…. and Answers!

Direct future questions to: Lee Barnes Founder and CTO Utopia Solutions, Inc. Email: [email protected] Twitter: twitter.com/USI_LeeBarnes LinkedIn: linkedin.com/in/leebarnes Blog: www.RaiseYourMQ.com

mailto:[email protected]