Sd times-june-24-2015

June 24, 2015

“Make Testing a First Class Citizen in your Development Process”Dan Boutin – Senior Product Evangelist

[email protected] (404) 304-9529@DanBoutinSOASTA

mailto:[email protected]

© 2014 SOASTA. All rights reserved. March 3, 2015 2

Agenda - Overview

o First, why is that important? What’s Missing?

o So, when did testing become second class?

o How do we find it?

o How do we keep it?

o Action Plan

3

Why is the “E” important?Performance is Everything

Revenue

Brand

Competitive advantage


So Where did the “e” go, anyway?

o It was first reported missing in 1995

o Actually dates back to 1989

o Testing has been more about:• Coding• Infrastructure & harness set-up• Logs & spreadsheets

o What is the “E”? Engineering!!


Performance In General – Where we lost the “e”

Linear approaches include delays between testing

UtUnit

Testing

FtFunctional

Testing

Development Testing Operations

PtPerformance

Testing

MoMonitoring

Legacy Approach: Silos and waiting

TIME

o Who’s the voice of the NFR’s? And, where?o Let’s look within the testing window to see what is really

going on


Retest

Process changes = new opportunities

Test Development Infra setupRepeat

?Fix

Test

Analyze

The Legacy Approach

TIME

Where we lost the “e” And why!

• Test “development” became its own SDLC• Complete with coding, peer reviews, etc.• Why? Script coding!

• Infrastructure “set-up” became expensive• …and did not scale to production• Why? Costs.

• Analysis was manual • Actual “testing” window was small• Still no NFR flow & traceability


So, how do we find the “e”?

o Performance Testing is NOT Performance Engineering

o Performance Engineering consists of many key processes

Modeling

Testing

ApplicationDevelopment Life Cycle

ITOperations

CapacityManagemen

t

PerformanceManagement

PerformanceEngineering


So, how do we find the “e”?

o Performance Engineering is all about the Analysiso It is not about “scripting” or “coding”o It should not be about infrastructure set-up


What has made the “e” so important again?


So, how do we get back the “e”?o Tear down the silos

• Take the discussion out of “testing” & put the focus in ENGINEERING.

• Embed performance engineering in each silo

• It starts in the Dev silo => Performance IS a design requirement

• In an Agile world, the UX IS all about performance


Testing in Production – Why Not?o “We don’t test in production.”

o This is a direct result of lack of visibility into actual test execution

o So what should the process look like?


Eliminating Delays Compresses TimeEnabling more testing to be done earlier and more often

UtUnit

Testing

FtFunctional

Testing

Testing

PtPerformance

Testing

MoMonitoring

Legacy Approach: Silos and waiting

Agile Approach: Automation and Continuous Testing

NFR’s built into the entire lifecycle

OperationsDevelopment

Dev OpsTesting

TIME


Continuous Delivery presents new opportunities

The ENGINEERING Approach

TIME

The Legacy Approach

Run more tests in less time.Start earlier.Cover more ground.

xn

/ TestAnalyz

e

Fix

Test Dev

Performance Engineering is Not a Singular Event

Test Development Infra setupTest

AnalyzeRetes

tRepeat

?Fix


Testing with Specific ObjectivesTests at different stages deliver higher coverage

Component scalability

Performance baseline

Integration testing

End to End w/ ServiceVirtualization

Unit scalability

Database testing

3rd party services

Live E2E tests

Stress testing Soak test

(reliability)

Failover

Capacity

Latency

PerformanceRegression (smoke tests)

Network

Testing (staging) Operations (prod)Development (sandbox)


Full Lifecycle Testing - Lab to ProductionTests at different stages delivers higher coverage

API Testing(REST/SOAP

& now WebSockets)

Testing (staging) Operations (prod)Development (sandbox)

Run smoke tests with

Continuous Integration frameworks

Run larger tests sooner with Virtual

Private Clouds

Introduce external

load sources with the

cloud

Test to beyond

peak expectation

s

Establish true user SLA’s

(and monitor them)

Directly test your

databases to scale

Test the “hard to

test”(Java custom

modules)

Let’s put it all together!


1. Build performance tests which will model real user behavior on your site

2. Tie business metrics (ROI) to website performance metrics and real user data

3. Correlate data analytics so you can troubleshoot bottlenecks and improve performance.

Now we tie it all together:

PerformanceAnalytics

Optimize performance across your online business

Requires a Continuous Approach

© 2014 SOASTA. All rights reserved. 19

Laying the FoundationA factory streamlines for quality and speed

Does it work as designed?

Does it perform under extreme

circumstances?

Approve materials Real Drivers

What happens in actual conditions?

Assembly Line

© 2014 SOASTA. All rights reserved. 20

CI streamlines for continuous quality

Functional validation

Performance,reliability,scalability tests

Real Users

Build

What happens in actual conditions?

Unit-tested code

Laying the Foundation


Continuous Quality

SOASTA’s Lifecycle

How we do it at SOASTA


Continuous Quality

I JUST MADE A CODE CHANGE

Jenkins runs any unit tests as usual



Continuous Quality


Run iterative load tests via Jenkins

Spin up suitable, consistent test environments in the

cloud or the lab





Continuous Quality



Mobile? Run functional automation on real devices for functional & performance validation


cloud or the lab





Continuous Quality



Daily (or more frequent) updates keep teams quality focused

Embedded failure details & bottleneck data isolate issues



cloud or the lab





Continuous Quality





Manage to a performance

baseline



cloud or the lab





Continuous Quality


Use real user data toiterate dev & test plans




Manage to a performance

baseline



cloud or the lab




CONFIDENTIAL – Not for Distribution © 2015 SOASTA. All rights reserved. January 13, 2015

Build performance tests that model user behavior on your site

First you need to determine the workload for the test.


How are companies without the “e” faring today?

What if I have a new process?


o What information I need?

● When is your peak second/minute/hour/day?● What is your peak volume? (active sessions, unique users, page visits, orders, ...)● What is the median session length?● What is the distribution of user paths?● What is the distribution of devices?● What is the distribution of locations?

o Good Test Data● Search Terms● Product Distribution● Browse Distribution● Cart Size● Order Size

How to get a realistic load testFactors to consider – Now that we have the “e” back!


Let your customers be your guide…


Analyze the most common session paths of real users


Test how you want, when you want, where you want


1. Let your users help you determine your performance goals

2. Tie business metrics, (conversion rates, length of engagement, etc) to calculate costs of performance vs revenue gain

3. Use data analytics to review site performance data to isolate bottlenecks, which can guide your performance investments

Summary & Takeaways


We will close with some Trivia

@DanBoutinSOASTA


Dan Boutin – Senior Product [email protected]

Mobile (404) 304-9529@DanBoutinSOASTA

oSOASTA Products

•FREE DownLoad: http://goo.gl/6CErMN

oVisit our Blog: http://goo.gl/TA2HKP

oQuestions? @DanBoutinSOASTA


http://goo.gl/6CErMN

http://goo.gl/TA2HKP

June 24, 2015

“Make Testing a First Class Citizen in your Development Process”Dan Boutin – Senior Product Evangelist

[email protected] (404) 304-9529@DanBoutinSOASTA


Technology

Sd times-june-24-2015