Cloud Based Test Automation

Fergal Hyneswww.ExpertSoftwareTest.ie

Goals & Context

Frameworks

Cloud Intro

Case Studies

Cloud Based Auto Test Execution

Presentation Structure

Our Goals Today

• Understand (some of !) what's possible re. testing on the cloud

• Presenter to avoid product placement and remain agnostic w.r.t. open source \ commercial

• Allow you to decide if cloud based test automation can help you

Context

Planning and Control

Analysis and Design

Test Implementatio

n and Execution

Evaluating exit criteria and Reporting

Test Closure activities

Frameworks

Framework 1(a): Test Cases \ Test Suites

• Focus on the E2E Processes

• Analyse and prioritise the E2E Business Processes

• Assign each test to either Smoke, Core or Full Test Suite

• Separate the E2E Processes out by functional area

Auto the ST first!

• Write the tests to test the E2E Business Processes

Framework 1(b): Test Suites \ Test Cases

Resultant Product Risk Profile:

Framework 2: Test Automation Project \ Scripts

Test Cases

Business Process Layer

Action Layer

Abstracted Data

Framework 3(a): Test Environment \ SW & ..HW

VCS

Version Control Solution Build Management Solution

Execution Management Solution

Reporting Solution

Execution Machines

Software Software

Software Software

Hardware

So…..

Get your 3 Frameworks Right First :

(a)Test Suite Framework(b)Test Auto Framework(c)Test Environment Framework

Therefore at this point you will have:VCS

OFFICE

CLOUD

The Reality…

Cloud

The Top 3 Advantages of Cloud Based Test Execution

• Saves Test Analyst Time

• Inherently Supports Scalability

• Is Cheap (no capex and low running cost if managed properly)

Cloud in 30 secs…

• Virtual machines in a specific remote location & managed via your web based account

Simple!

• Connect to machines via (for example) RDP for windows and putty \ SSH for linux

• File transfer by (for e.g.) FTP

Clarification

The Test Machines

The Product

Machines

Test Automation on the Cloud 1

The Test Machines

• Security Group• Each orange box ~ 1 or >1 Machines• VM management (e.g. create \ destroy)

Test Automation on the Cloud 2

The Test Machines

The Product

Machines

• Product can be cloud or not cloud!• Trigger on event\schedule\on-demand

Case Study 1 – API & Mobile Apps

• Microsoft house• Locally Hosted API (i.e. Non Cloud based )• Event Driven and On-Demand Functional and Non-Functional Test Exe

Case Study 2 – API & Web

• Open source house• 100% Cloud based product under test• Functional and Non-Functional Test Exe On Demand

Special Note: Non-Functional Testing(Perf, Load, Stress)

• Regional load generation capability supported on cloud• SAAS

The Top 3 Pros \ Cons of Cloud Based Test Execution

Priority Pros Cons

1 Time. Backup, Reuse, Version Control via VM management Console

Your company and it’s security policies need to be flexible. Over to you…

2 100% Scalable As secure as you are - Choose your security level carefully – Broadcasting is rife

3 Cost – No capex and environment prototyping is essentially free. Billing reports are good.

Beware of Mobile Device Farms – Reliability and Cost(Can be discussed later if you want )

More Pros \ Cons of Cloud Based Test ExecutionPros Cons

Personal computers, internal and public internet networks can be abstracted from the test execution environment avoiding latency and local\network failures and false negatives

Domain knowledge inertia

Accessible anywhere - Walk away, interrogate results\progress from home

Cost can spiral if not controlled and monitored

Machine resources are configurable (almost) on the fly

Images (backups) become unsupported over time

Regionally distributed worldwide iOS not well supported

No physical machine maintenance overhead

Ability to share machines between cloud accounts

Provides many useful facilities, e.g. machine resource monitoringPlatinum services available for enterprise applications

Drives Continuous Integration best practices (local dev and remote execution)Not much you can't do...

Main Takeaways

3 Key Takeaways

Get your 3 Frameworks Right First :

(a) Test Suite (b) Test Auto(c) Test Env

Prototype your Test Auto

Solution on Free Machines on

your Cloud Account

If you do nothing else…..get the following right:

Idealfor

Perf, Load and Stress Testing

!

CONTACT US !

Expert Software TestDalys Lane, Main Street, Killorglin, Kerry, Ireland.V93 RT2P

(066)9796644fergal@expertsoftwaretest.ie

Q & A!