uTest Whitepaper Load Testing for Less

7/28/2019 uTest Whitepaper Load Testing for Less

http://slidepdf.com/reader/full/utest-whitepaper-load-testing-for-less 1/11

Load Testing For Less 7 tips on affordable load testing for startups and SMB

WHITEPAPER: May 2010



WHITEPAPER: Load Testing For Less 1

Table of Contents

Load Testing: The Breaking Point for Software Companies………………………………………………….… 2The 'Peak Performance' Dilemma …………………………….………………………………………………………………..…………………….………………..…. 2

New Approaches………………………….………………………………………………………………………………………..………………………………………………….…………. 2

Load Testing Fundamentals…………………………………………………………………………….………………………………………………….……………………….. 3

Software Stress Testing………………………………………………..……………………………………………..…………………….………………………..……. 3

Hardware Stress Testing……………………………………..…………………………………………………………….………………………………………..……. 3

Soak Testing ……………………………………………………………..………………………………………………………..………………………………………………..……. 3

Spike Testing ……………………………………………………………..………………………………………………………..………………………………….…………..……. 3

Configuration Testing ……………………………………………………………..………………………………………………………..………….……………..……. 3

Platform Testing ……………………………………………………………..…………………………………………………………………………..…………………..……. 3

Load Testing For Less………………………………………………………………………………….……………..…….… 4

Simulated Load Testing ………………………………………………………………………………………………………………..………………………………….…………….. 4

Load Testing Tools……………………………………………………………………………………………………… 4

Common Scenarios………………………………..……………………………………………..…………………….………………………..………………….…………. 5

Live Load Testing ………………………………………………………………………………………………………………..………………………………………………………..…….. 5

Common Scenarios……………………………………………………………………………………………………. 5

Hybrid Load Testing ………………………………………………………………………………………………………………………..……………………………….……………….. 6

Common Scenarios……………………………………………………………………………………………………. 6

Load Testing Tips………………………………………………………………………………….……………...………….… 7

Test What Matters Most …………………………………………………………………………………………………………..………………………………………………….. 7

Redevelope The Metric System…………………………………………………………………………………………… 7

Watch and Listen To Your Results ………………………………………………………………………………………..……………………………………………….. 7

Test In The Wild………………………………….………………………………………………………………………………. 8

Know What You're Looking For ……………………………………………………………………………………………………………………………………………….. 8Don't Test Too Much (or too little)………………………………..……………………………………………………. 9

Be Environmentally Friendly……….………………………………..……………………………………………………. 9

About uTest……………………………………………………………………………………………….………………….… 10

“It's easy to cry "bug" when the truth is that you've got a complex system and

sometimes it takes a while to get all the components to co-exist peacefully .”

~ Doug Vargas




Load Testing: The Breaking Point for Software Companies

The ‘Peak Performance’ Dilemma

At what point will your application’s performance begin to degrade? How many concurrent users can itsupport? How do the JavaScript-based components of your website behave with 50, 100 or 1000 active

users? Where are the bottlenecks between your code base, database, CDN and load balancers? How is

all this affecting the user experience? And your page views, bounce rates, and ultimately, your revenue?

Answers to these vital questions - questions that cause even the most seasoned tech execs to lose sleep

(and sometimes their jobs) - are determined by comprehensive load and performance testing. The

problem here, especially for start-ups and small businesses, is that traditional load testing has proven

to be prohibitively expensive and requires specialized skills , as even base packages can range from

$20k to $50k.

Unfortunately, far too many companies reach a point where the benefits of load testing are outweighed

by the costs - hence the dilemma of testing and attaining peak performance.

New Approaches To Load Testing

Innovations in crowdsourced testing and open-source tools have resulted in a load testing methodology

that is far more adaptable to growing software companies. Startups and small businesses now achieve

load testing coverage to match that of most large enterprises – and they do it by combining real-world

testers with the latest in automated technology.

This brief whitepaper will outline the basics of affordable load testing. While discussing how this

emerging methodology works, it also will address the fundamentals of load testing, including tips on:

Targeting core features: Learn how to prioritize the most vital features of your website,

especially when developing test scripts and common user scenarios.

Defining key metrics: How should you gauge download speed? Active server pages? Processing

time? We’ll show you how to define key metrics and avoid needless analysis.

Load testing frequency: How often should your company run a load test? What are the big

events that make it a necessity?

Environments: Should you run load testing on a staging environment or on production? We’ll

outline the benefits and danger of each approach.

And more: Successful load testing requires the right combination of technology, people and

processes – and we’ll show you the best way to leverage them each step of the way.

http://www.utest.com/







Load Testing Fundamentals

A unique characteristic of load testing is that it’s performed for many different reasons. Though it’s

sometimes viewed as companies simply wanting to “break things” software companies rarely have thesame objective.

Here are a few more common load testing terms and scenarios:

Software Stress Testing: In this instance, a company is looking to determine where the software

begins to break down under extreme load. Simply put, it is people wanting to “break things.”

Hardware Stress Testing: Similar to software stress testing, this procedure performed to

determine where a benchmarked set of hardware starts to break down under extreme load. A

company, in this instance, seeks to ensure that their system is robust under normal usage.

Soak Testing: This is a simulation that often takes place over the course of days or weeks thattests for performance degradation caused by things like slow memory leaks or even something

as simple as hardware fatigue or subpar hosting conditions. It focuses on long-term usage.

Spike Testing: This is performed to determine scalability under a sudden increase in load. The

ability of an application to shift gears and go from handling a moderate number of users

efficiently, to handling a large number of users adequately is of course very important. If your

company makes the front page of a news website or an aggregate like Digg or Slashdot – and

your normal load spikes heavily – you will want to be prepared.

Configuration Testing: Here a company would want to compare relative performance based on

sets of server parameters. If a company is unsure of what hardware and software options are

best for their application, they can run a series of tests with the exact same user load, andmonitor the relative performance in each scenario.

Platform Testing: This procedure is geared towards companies that want to test end-user

functionality on a broad spectrum of hardware and software. This allows them to test their

software's front end under a wide variety of hardware and software configurations. This helps

them determine their software's requirements, which they can publish as their suggested

minimum configuration.

We will now examine methods that will provide unique insight into your application, no matter what

type of load testing you plan on performing…




Load Testing For Less

Simulated Load Testing

There are a number of simulated load testing tools on the market today - both commercial and free. Of

course, there are many advantages to using the commercial tools, as they tend to have cleaner UI’s,

more efficient reporting tools and documentation, to name a few perks. However, be careful not to

discount the impact of having an entire open-source community reviewing the code for bugs, making

contributions, and making sure that the testing tools are spyware-free.

That said, here’s a short list of tools you may find helpful (listed alphabetically):

BrowserMob – Free Load Testing Tool

BrowserMob uses real browser users for website load testing, providing better AJAX support,

clearer reporting of errors and an experience that more accurately mimics that of the real user.Run concurrent tests on your website with real browsers, not simply HTTP requests, and catch

errors that other low cost providers miss.

The Grinder – Generic Jython Load Tester

The Grinder uses the Java implementation of Python (Jython), which is a script-only, feature rich

load testing platform. It is possibly the most powerful lightweight free load testers on the

market, but also the most difficult to master. No GUI to speak of, very little in the way of

examples, but it has a terrific FAQ. It isn't useful for front end automation, but otherwise The

Grinder is flexible and infinitely useful if you can master it. This is the programmer's preferred

load testing tool.

JMeter – Graphical Load Tester

JMeter is without a doubt, the most downloaded free load testing software on the planet. It has

a GUI written in Java, which means you can use it on just about any operating system, so your

testing is truly portable. It has a flexible scripting language, as well as plugins for all sorts of

strange test scenarios. Best of all, it is one of the most completely documented load testing

applications out there, with ample tutorials and great community support. It is useful for just

about every test type you'll eventually run into.

OpenSTA - HTTP and HTTPS Heavy Load Tester

OpenSTA is one of the most flexible load testing tools on the market today. Not only can it use

recorded scripts and play them back, but it can convert the recorded macro into a powerful

script that is easily edited and injected as a hypertext stream. OpenSTA is therefore best of both

worlds, and is suitable for just about every form of load testing.

SOASTA – Leader in Cloud Testing

SOASTA leverages the cloud to offer on-demand load & performance testing, enabling you to

test the real-world performance of your web apps. This functional testing can either be in

separate from SOASTA’s load testing runs, or performed while an app is under simulated load to

test how it works under pressure.

http://browsermob.com/performance-testing


http://grinder.sourceforge.net/


http://jakarta.apache.org/jmeter/


http://opensta.org/

http://opensta.org/

http://www.soasta.com/



http://opensta.org/







Of course, these are just a couple of tools that you should have on hand for general debugging. Your

operating system will have basic network usage monitoring, so you can see if you're having any sort of

bandwidth issues on your end. But there is no better network interface snooping tool than Wireshark,

so make sure you have it on hand, just in case you need to debug the IP stack.

Common Scenarios for Simulated Load Testing

Simulated load testing services are typically leveraged by companies that meet the following criteria:

High-Traffic Apps: If you're expecting a lot of concurrent visitors or users, this method can

provide you with hundreds of thousands of simulated users - helping you to identify any load-

related weaknesses and assure performance during peak usage.

Fast-Approaching Deadlines: Most automated tools require weeks of planning and on-ramping.

Using the tools and methods mentioned above, companies can set up and execute a simulated

load testing cycle in a much shorter time span.

Changing Apps: Recent changes to any link in your infrastructure “chain” (i.e. features,functionality or even content)

Limited Internal Resources: Traditional load testing tools or services often burden in-house

resources, especially QA and IT staff. These tools eliminate much of the heavy lifting, letting you

concentrate on more urgent matters.

Live Load Testing

Unlike the great majority of load testing methodologies, this procedure requires NO automated or

synthetic testing tools. Instead, a team of real live testers will converge on the core features of your

application simultaneously . Thanks to advances in crowdsourced testing, it has become extremely easy

to engage a global community to quickly converge on whatever features you deem most important.

Logging bugs and usability issues, live testers can provide an important new layer to your metrics. Andbecause they’re professional testers, they know what to look for and how to report/document issues

that they may encounter.

Furthermore, many startups and small businesses - particularly B2B companies - aren’t the least bit

concerned with preparing for 500,000 or a million simultaneous users (though it would be nice). Instead,

they would much rather know whether or not 200 live users can simultaneously complete a checkout,

sign in to their account, and perform other key actions without any noticeable deterioration. In other

words, they want to test against real-world conditions.

Common Scenarios

Since simulated load tests only tell part of the story, here are a few situations where live load testing

makes the most sense:

Flash and Multimedia Apps: When launching a web app that incorporates flash, video and other

streaming media, your automated tests may indicate that all is well, even when it's not. Live

testers, on the other hand, can determine if the sound quality is off, if the video is pixilated, the

file takes too long to buffer or other problems that simulated load testing rarely catches.

Limited Traffic Apps: As mentioned earlier, real-world conditions might not entail hundreds of

thousands of users. Most load testing tools require too much effort and incur great costs to

http://www.wireshark.org/







leverage them for such a comparatively small assignment. If your app has moderate traffic – and

this is part of your business model – it is much better to gauge how real visitors will experience

your application.

Startups and SMBs: For most small and medium-sized companies, traditional load testing tools

are seen as overkill, not to mention time-consuming and requiring specialized skills. Forgoing

such tools and relying on the expertise of professional testers enables you to simulate the

optimal amount traffic in a quick, easy and cost-effective manner.

End-to-End Load Testing: For companies that want ‘in the wild’ testing that mirrors the TRUE

user experience – and to avoid complicated automation – live load testing is an ideal solution.

Hybrid Load Testing

Combining live testers with simulated load tools, hybrid load testing offers greatest insight into your

application’s performance. Here, a team of testers will execute test scripts and perform common actions

on the front end of the software, while automated tools place your application under synthetic load.

Rather than relying entirely on backend automation, real testers –geographically dispersed – can verifyFlash-based components, cross-browser performance, feature sets, page load times and other functions.

Common Scenarios

Hybrid load testing is employed by companies that seek to maximize testing in the following areas:

Flash and Multimedia Apps: For companies that need complete load testing, but also want to

verify that the Flash, video and other multimedia components of their web application are

working properly with no degradation under peak load, hybrid testing makes the most sense.

Load-Related Defects: Some bugs – notably GUI bugs – only show themselves when a web

application is under heavy stress. Hybrid testing enables companies to discover these defects

through first hand bug reports. Load-Related Feedback: Through hybrid testing, companies can sort through the data coming

from synthetic load testing, while also gathering real-world feedback from software-savvy users

all over the world – adding an entirely new layer of performance-related feedback.

And now, the seven tips for affordable load testing…




Tips For Affordable Load Testing

Test What Matters Most

What are the most important actions your typical end user will take? What features will have thegreatest negative impact if they fail? This process should not be taken lightly. For many companies, this

step will include a thorough check of e-commerce features such as online checkouts, shopping carts,

order forms and other features. If these actions are not so clear, begin by examining your web traffic

through Google Analytics or something comparable.

Redevelop the Metric System

Page load speed isn’t an issue for everyone. For others, it is a matter of life and death (financially

speaking anyway). So while it’s easy - and perhaps necessary - to dwell over active server pages,

requests per second ratios and other technical indicators, the data most worth analyzing is that which

directly involves your end user, so it would be wise to tier your metrics accordingly.

Here are a few suggestions for server-based load testing metrics:

• Number of users and/or sessions

• Average session time

• Number of page views

• Average page views per session

• Peak period

• Number of hits

• Average page size

• Most requested pages

• Average time spent on page

• New users vs. returning users • Frequency of visits (e.g., 75% of users made one visit)

• Demographics

• Browser, browser version, Java script support, Java script enable/disable, etc.

Watch AND Listen to Your Results

When running simulated load testing, remember that such virtual systems rely on emulators for video

and audio output . Also, it does not consume all of the RAM, but only a part of it. So it can be extremely

confusing when you stated in your

system configuration that your PC has

2GB RAM on board, but in fact your

virtual machine is configured to use

only 512MB. Realize that when it

comes to audio and video, real users

are essential.




Test In the Wild

Most likely, you didn’t build your application for automated users. You built it for real users. Why then

do so many companies rely solely on automated testing results to verify functionality under stress? Until

recently, it was because there was no such mechanism by which a company could utilize real testers

while a system was under load. But as we outlined in the Hybrid and Live Load testing section of this

whitepaper, that obstacle has been overcome. By leveraging real testers with automated tools,

companies now have access to entirely new layer of data. Better yet, they are able to identify and

resolve issues that would have been reported later in the development process.

Know What You’re Looking For

For an effective load test – and to

avoid wasting thousands of dollars in

the process – it’s essential that you

know what you need to get out of it . Itmay seem obvious, but it’s a routine

mistake, especially for startups and

small businesses. To reach any

actionable conclusions (which is the

entire point of load testing) companies

must be able to identify where specific

failures occurred in your delivery

chain. Did the major failure first occur with your database or code? Or did the issues originate from your

web server, load balancer, content delivery network or ISP? Companies that fail to pinpoint these issues

will learn a valuable lesson in the costs of redundant testing. Don’t be one of them.

(Sample chart from a real-world load test, with each line representing different transactions within the same app.)




Don’t Test Too Much (or too little)

How often should companies run load testing of their application? Of course, this answer will vary from

company to company, but generally speaking, these are ideal times for startups and small businesses to

consider load testing:

Major product launches or new features

Changes to your cloud configuration

New servers, content delivery networks, etc.

Significant spike in users and/or traffic (+50%)

Be Environmentally Friendly

One of the more obvious questions to arise when preparing to run a load test is, “Should I test on

production?” Again, there is no right or wrong answer. While it is certainly helpful to have a staging

environment that is identical to your production, it is not an absolute necessity. But to prevent wrecking

your production site, it is helpful to start small with your load testing efforts, as most defects arediscovered by low-volume tests (and by live testers). Once these issues have been identified, you can

then move on to more comprehensive load testing. At that point, it will be more helpful to have a

staging environment.




*Special thanks to uTester Bill Ricardi for his contributions on load testing and performance*

About uTest

Headquartered near Boston, uTest is the world's largest marketplace for software testing services. The

company provides real-world QA services – including simulated, live and hybrid load testing services –

through its community of 25,000+ professional testers from over 160 countries around the world. To

date, more than 500 companies - from web start-ups to enterprise software firms - have signed up to

get their apps tested by the uTest community.

uTest enables companies to launch higher quality products; get their desktop, web and mobile

applications to market faster; and control the cost of testing. Customers specify their requirements for

tester experience, location, language, OS and browser, and uTest selects the right testers for each

project. And because uTest is on-demand, companies only pay for completed test cycles.

A brief online video is available at www.utest.com/intro. uTest can be contacted at:

uTest, Inc.

153 Cordaville Road

Southborough, MA 01772

p: 1.800.445.3914

e: [email protected] w: www.utest.com

http://c/Users/mikeb.CORP/AppData/Local/Microsoft/Windows/Temporary%20Internet%20Files/Content.Outlook/Z6ZRF7TW/www.utest.com/intro



mailto:[email protected]









Documents

uTest Whitepaper Load Testing for Less