Testing in the Cloud

An investigation of the cloud platform for performance testing

Richard BishopSenior Performance Consultant - Intechnica

Richard Bishop

Senior Performance Consultant at Intechnica 18 years working in IT 13 years as a performance tester

• Acuma (consultancy), Cisco, BT, Vodafone, HBoS, Barclays• HBoS, LBG• Independent contractor• Intechnica

Involved in Mercury User Group Now “Chapter leader” and director for Vivit, HP Software User Group

Contents

Introduction Cloud introduction Cloud hype Common concerns

Cloud Testing Cloud-based testing Pros and cons Choosing a test tool Testing in and from the cloud

Cloud Introduction

“Cloud computing is a model for enabling convenient, on-demand

network access to a shared pool of configurable computing

resources that can be rapidly provisioned and released

with minimal management effort or service provider interaction.”

Source: US National Institute of Standards and Technology (NIST)

Cloud does NOT equal simplification...

The Hype Cycle

As defined by Gartner, the Hype Cycle describes how new technology goes through a cycle of hype and expectations

before its true potential and benefits are eventually realised.

The Hype Cycle

The Hype cycle in detail...

The Hype Cycle - 2011

Cloud types

Cloud is a very broad term and it makes sense to subdivide it further.

In traditional IT, you manage entire stack. Apps, Data, Middleware, OS, Physical hardware etc…

Cloud types

IaaSResponsibility for raw block storage, networking and hardware is outsourced.

Crossover point is at OS level.Utility computing model, pay for what you use.

Cloud types

PaaSResponsibility for support and upgrades of operating

system is also passed over to third party.Similar “pay for what you use” computing model.

Cloud types

SaaSComplete application, data, responsibility for management, maintenance etc.

are passed to a third party.

Cloud types

Common concerns

All “eggs in one basket”

Unproven technology

Support / Learning curve

“Lock-in” (e.g. Azure / AWS proprietary databases)

Costs difficult to quantify and control

Data security and loss of control

Performance, performance, performance……..

vs.

Cloud-based testing

Where does each test tool fit in the cloud?

Cloud-based testing

Cloud test pitfalls

http://www.infoworld.com/d/cloud-computing/face-the-facts-cloud-performance-isnt-always-stable-170066

“Cloud computing performance varies more than you might think….”“….. the price of consistency likely exceeds what you want to spend”

http://www.twitter.com/DavidLinthicum

Cloud test pitfalls

“Cloud computing performance varies more than you might think….”“….. the price of consistency likely exceeds what you want to spend”

This is true for testers as well as developers – we need to work around cost uncertainty & variable performance of cloud platforms.

Measure the performance and consider provisioning some load from conventional sources too!

Realism vs. Repeatability

Realism Repeatability

But how will itreally perform?

Did my change make a difference?

As testers, need to balance realism and repeatability.

Different types of tests require different approaches

Our approach to cloud-testing

Avoid pitfalls• Repeat tests to obtain average results• Avoid peak load times for cloud environments• Re-validate tests within physical test environments• Use diagnostics software to drill down into application performance

…..better application understanding is vital

Intechnica’s tool, based on Facilita Forecast, for flexible load & stress testing from the cloud.

Our approach to cloud-testing

Use benefits of the cloud• Cloud-based load generators• Cloud-based test environments• Cloud-based test environment management

Intechnica’s tool for provisioning cloud-based environments with simplified interface & startup/shutdown scheduler – Available now

Test tools sample checklist

Platform support Ease of installation Ease of scripting Ease of use Data management Cost Stability Scalability Repeatability Market penetration

(availability of skilled testers) Support

Real-time monitoring Real-time load adjustment Dynamic load-adjustment SUT monitoring

o e.g. UNIX / Windows support

Integration with other toolso e.g. Wily, dynaTrace, HP

diagnostics Limits on load (licence / physical) Scheduled tests IP spoofing Rendezvous points Bandwidth throttling

The Cloud Testing Future

Traditional computing models will increasingly be replaced by cloud

Migration can’t be done without careful planning

100% cloud isn’t desirable for many sectors• Mixed cloud based and hosted load generation• Cloud based and traditional test environments• Hybrid approach is more likely

Doesn’t have to be “all or nothing”

Cloud works • Only if you plan and develop performance tests to exploit its benefits• Key thing is to balance realism and cost

Web: www.intechnica.co.ukEmail: more@intechnica.co.ukTel: 0845 680 9679Fax: 0845 2991647Address: Fourways House, 4th Floor, 57 Hilton Street, Manchester, M1 2EJ

Contact

more@intechnica.co.ukhttp://www.intechnica.co.uk/@intechnica