DevOps Guide

Performance Testing Dictionary

A QUICK GUIDE TO BRINGING YOUR TESTING VOCABULARY

INTO THE 21ST CENTURY

SOASTA: Performance Is Everything 2

In the world of performance testing, a lot has changed in the past 25 years.

Today’s testing solutions have evolved light years beyond the tools that were developed back when Dallas was a city AND a TV show. That’s why we created this dictionary — to translate your circa-1989 client/server tool terminology to today’s web and mobile cloud-based performance testing language.

If you’re using — or planning to migrate to — SOASTA CloudTest® from an older tool like HP LoadRunner, this dictionary is for you. Our goal is to help make your transition to modern performance testing as seamless as possible.

CLOUD TESTING TERMINOLOGY AND CONCEPTSBoth CloudTest and LoadRunner require planning before creating performance test cases. The performance engineer needs to understand the system, user experience, and business drivers for load testing the system under test. The load test plan needs to identify test cases, Entry and Exit criteria, metrics to be collected, timelines, and include a final report.

Here’s an overview of some commonly used terms in both CloudTest and LoadRunner, with additional explanation on the following pages.

CloudTest LoadRunner

Environment Environment

Maestro Load generator

Virtual users Vusers

Clips Vuser scripts

Composition Scenario

Conductor VuGen

Session template package wizard Scan script for correlations

Seed data repository Parameterization functionality

Validation Content check

Transaction / collection Transaction

Analytics dashboard Performance analysis

Repository Result directory

SOASTA: Performance Is Everything 3

ENVIRONMENTSAn environment in CloudTest is very similar to an environment in LoadRunner. They are both web applications. Both perform the job of controlling virtual users for a test scenario, and both require a username and password to access the Performance Center manages and maintains scenarios.

LoadRunner Controller, the stand-alone solution, also does this.

With SOASTA, it is done by SOASTA’s repository in CloudTest. One important difference is that the SOASTA CloudTest environment incorporates load test creation, browser-based functional test creation, and can serve as a load generator, all in one platform.

MAESTROA maestro in CloudTest is the same as a load generator in LoadRunner.

In the case of SOASTA CloudTest, a load server can be local machine or it can be in the cloud. Deploying load generators in the cloud is cost effective and can simulate traffic from different geographic locations on the Internet.

With LoadRunner, load generators are usually deployed locally. A recently announced version of LoadRunner provides the ability to deploy both controller and load generator in the cloud, but this storm is still evolving and does not yet qualify for a name from the National Hurricane Center.

CLIPS AND VUSER SCRIPTSClips in CloudTest are similar to Vuser scripts in LoadRunner. A composition in CloudTest is similar to a scenario in LoadRunner. Both composition and scenario determine what should happen in a testing session with details about what test cases should be

BUILDING SCRIPTSLoadRunner uses VuGen. VuGen captures all the recorded traffic based upon the type of protocol selected for the test.

With SOASTA CloudTest, recording is integrated in the product and uses an agent called the SOASTA Conductor on the machine from which recording is being done. The Conductor acts as a web proxy for capturing all the HTTP/HTTPS traffic when recording. SOASTA CloudTest has the option of viewing recording in list and icon views.

SOASTA: Performance Is Everything 4

SEED DATASOASTA CloudTest has a seed data repository, which is similar to LoadRunner’s parameterization functionality, but covering only the parameter type File. Test data can be stored within CloudTest or it can be hosted anywhere accessible by the load generators and used within a clip. Once test data is available, it can be used in a random fashion, as single shot data or unique data for each virtual user. This functionality can be achieved either from the UI or by using JavaScript.

CORRELATIONSOASTA CloudTest has a Session Template Package Wizard, which is somewhat similar to LoadRunner’s Scan Script for Correlations. However, SOASTA CloudTest scans for name/value pairs and not for differences between recording and playback. The Session Template Package Wizard identifies all dynamic values in the clip and required values can be selected from the UI.

SOASTA: Performance Is Everything 5

COLLECTIONSCloudTest can group multiple messages or pages as transactions (also known as collections) and metrics can be tracked in the analytics dashboard for a particular transaction. This is similar to the concept of transaction in LoadRunner.

VALIDATIONCloudTest has validation functionality similar to LoadRunner’s Content Check. The validations can be done within a clip/script and added for text, HTML, JSON, XML and SOAP. Validations can be done for data within the header or body. The validations can be added for each and every message within a clip. Similar to the content check functionality in LoadRunner, successand failure messages can be added as part of validations.

SOASTA: Performance Is Everything 6

ANALYTICSSOASTA CloudTest has extensive metrics in the analytic dashboards. These dashboards show real-time data, independent of the scale of the test. They are similar to LoadRunner’s performance analysis. SOASTA CloudTest results can be viewed during and after the test execution and results can be exported in .csv, .xml and .doc formats.

TEST COMPOSITIONSTo create a test in CloudTest, one or more clips are added to a composition. This is similar to adding vu scripts in a scenario. The composition specifies load servers, number of virtual users, number of iterations, ramp time and duration of the test. This is like adding load generators, building a schedule, specifying duration and ramp up and ramp down for the test in LoadRunner.

SOASTA: Performance Is Everything 7

REPOSITORYThe SOASTA repository stores results from the test. This is similar to the result directory in LoadRunner. The SOASTA repository can be maintained on premise or in the cloud and can accessed over the internet. In both cases, the client is an Ajax-based browser.

MONITORINGJust like LoadRunner, SOASTA CloudTest has integrated monitors for monitoring system utilization of systems under load. These monitors can be created as web monitors, database monitors, application server monitors, etc.

CloudTest can also integrate monitored data from third-party monitoring tools, such as those from New Relic, CA, Amazon, AppDynamics, Correlsense and more, displaying the results in the same real-time dashboards.

SOASTA: Performance Is Everything 8

Dan BoutinDan is the Vice President of Digital Strategy at SOASTA, where he is responsible for the market development of the world’s first Digital Operations Center (DOC). Dan has more than 15 years experience in the performance testing space.

@DanBoutinSOASTA

SOASTA: Performance Is Everything

We help companies like Apple, Target, Etsy, and Microsoft continuously

monitor, test, and optimize their web and mobile applications.

We can help you, too.

Sign up for CloudTest Lite IT’S FREE!