W11 Performance Testing

5/1/2013 1:45:00 PM

Cutting-edge Performance Testing on

eCommerce Websites

Presented by:

Ron Woody

GSI Commerce

Brought to you by:

340 Corporate Way, Suite 300, Orange Park, FL 32073

888-268-8770 ∙ 904-278-0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com

Ron Woody

Senior manager of QA performance at GSI Commerce (an eBay company), Ron Woody manages a team of twenty-one performance engineers dedicated to ensuring that web-based performance is within targeted requirements for all client sites and launches. Ron brings more than fifteen years of automation, performance, QA management, and process experience to his current role. He heads the effort to improve performance for all web stores released and supported by GSI while providing performance testing and engineering vision to the company. Ron has served as a consultant to companies such as Comcast, Bloomberg, Vanguard, and Bank of America.

4/16/2013

1

| 1

Automating ClientAutomating Client-- and Serverand Server--

Side Web Performance Testing Side Web Performance Testing

and Optimizationand Optimization

Ron Woody

Director of Performance

GSI Commerce, an

eBay Inc. company

| 2

WhoWho iis GSI Commerce?s GSI Commerce?

4/16/2013

2

| 3

About About GSI CommerceGSI Commerce

■ Companies of all sizes have selected GSI

Commerce® to help deliver an exceptional

and engaging eCommerce shopping

experience to their customers.

| 4

GSI Commerce: QA Performance ProcessesGSI Commerce: QA Performance Processes

■ Performance Testing

– Platform/Product

– Projects

– Store Launches

■ Client-Side Testing

■ Server-Side Testing

■ Performance Analysis

4/16/2013

3

| 5

Business Impact of PerformanceBusiness Impact of Performance

■ Performance impacts sales!

■ Poor performance increases the risk of site

abandonment

■ Slow performance may impact brand– People remember their experience when a site is

slow; Users may be reluctant to return

■ Our clients’ competitors are only a click away

| 6

Recent Performance Testing ObjectivesRecent Performance Testing Objectives

■ Phase 1: Improve the end-user experience

via client-side and server-side testing

■ Phase 2: Increase productivity via Automation

■ Phase 3: Create more realistic client-side test

scenarios and enhance our testing further

4/16/2013

4

| 7

Phase 1 Phase 1 –– Improve endImprove end--user experience via user experience via

clientclient-- and serverand server--side testingside testing

■ Client-Side Testing

– dynaTrace AJAX Edition

– Improve the End-User Experience

■ Server-Side Testing

– dynaTrace Development & Test Center Editions

– Diagnosing server-side bottlenecks

■ End-to-End visibility

| 8

ClientClient--Side TestingSide Testing

■ Started with dynaTrace’s free AJAX Edition

– More accurate rendering times; greater visibility

4/16/2013

5

| 9

ServerServer--Side TestingSide Testing

■ Added dynaTrace Development Team & Test

Center Editions

| 10

Phase 2 Phase 2 –– Increase Productivity via Increase Productivity via

AutomationAutomation

■ Server-side Automation

– LoadRunner with dynaTrace Test Edition

– dynaTrace AJAX Edition + Test Center Editions

provide in-depth views

■ Client-side Automation

■ Use of dynaTrace AJAX API’s to extend our

reporting

4/16/2013

6

| 11

Benefits of using dynaTrace: VisibilityBenefits of using dynaTrace: Visibility

■ dynaTrace provided visibility we hadn’t

had before

– But testing browser use-cases was manual,

and very time-consuming

■ Increasing Productivity via Automation

– LoadRunner / Performance Center

– QuickTest Professional / Selenium

– Quality Center / ALM

| 12

Solution: Automated Browser LabSolution: Automated Browser Lab

■ Virtualized browser lab to capture the end-user

experience with the touch of a button

■ Cutting-edge automated home-grown solution

– dynaTrace AJAX Edition

– Selenium

– Google Page Speed

– Webpagetest.org

– Shunra network emulation tools

■ Real-Time reporting system to show results

4/16/2013

7

| 13

GSI Commerce’s GSI Commerce’s Browser LabBrowser Lab

| 14

GSI Commerce’s GSI Commerce’s Browser LabBrowser Lab

4/16/2013

8

| 15

Why dynaTrace?Why dynaTrace?

■ Visibility into Rendering

– As well as other Web 2.0 challenges

– Was not available in HTTPWatch

■ IE 7,8,9 support

– Firebug didn’t support IE

■ Now dynaTrace supports Firefox as well

| 16

Basic Basic

Browser Browser

Lab Lab

DiagramDiagram

4/16/2013

9

| 17

GSI Commerce’s Browser Lab FrontGSI Commerce’s Browser Lab Front--EndEnd

| 18

A Sample Browser Lab ReportA Sample Browser Lab Report

4/16/2013

10

| 19

A Sample Browser Lab ReportA Sample Browser Lab Report

| 20

Using Automation to Save Time and MoneyUsing Automation to Save Time and Money

■ With automation, GSI saved hours per test

– Reducing test time from 20 hours to ~1 hour

■ Nearly a 20x time savings

■ Reproducible results

■ Shortens project time

– Allows for more iterative testing

4/16/2013

11

| 21

Additional Benefits of our Browser LabAdditional Benefits of our Browser Lab

■ Pinpoint problems quickly

– Problems that were difficult / impossible to see

before

– Full Waterfall / timeline

– Rendering

– DOM Manipulations

■ Allows us to be proactive

■ Leads to a faster site

■ Enables clients to make informed decisions

based on the results of our testing in QA

| 22

Benefits of dynaTrace: Better CollaborationBenefits of dynaTrace: Better Collaboration

■ Dev and QA collaboration

– dynaTrace AJAX Edition sessions shared with

Development

– Now Development sees exactly what Test sees

– Less back & forth

– Faster resolution / optimization

■ Dev Teams can now use dynaTrace prior to QA

■ Developed Performance Requirements that

include dynaTrace rankings

– Dev teams now design to these requirements

4/16/2013

12

| 23

Phase 3 Phase 3 –– Create more realistic clientCreate more realistic client--side side

test scenarios and enhance our testing furthertest scenarios and enhance our testing further

■ Network emulation for more realistic traffic

■ Larger samplings with automation

■ Browser weighting to generate more accurate

timings across browsers

■ Automatic removal of outliers

| 24

Performance Objectives Revisited (ROI)Performance Objectives Revisited (ROI)

■ Phase 1: Improve the end-user experience

– ROI: We now have visibility into performance issues and how to

resolve them quickly and more efficiently thus saving us time and

money

■ Phase 2: Increase productivity via Automation

– ROI: ~20x savings with our Browser Lab solution

– can now do testing and analysis simultaneously

– faster time to market; more complete testing

■ Phase 3: Create more realistic test scenarios

– ROI: More accurate timings through browser weighting and network

emulation; increased consistency by reducing noise in our results

4/16/2013

13

| 25

Recent additionsRecent additions

■ Leveraging dynaTrace browser agent (AJAX

Edition Premium Version)

– Ability to see end-to-end performance in QA

(PurePath) via Performance Center scripts

■ Use of dynaTrace (Production Edition) to

become even MORE proactive

– Ability to find issues that exist in Production

(including issues that clients haven’t reported yet)

– Determine where users are abandoning our sites

– Ability to monitor Apache

| 26

What’s Next?What’s Next?

■ Browser Lab aspirations

– Mobile testing

– Cached vs. Non-Cached pages

– Film strip

– Integration with webpagetest.org

■ Expand to Continuous Integration

– Enable Dev to find/fix problems before they reach

QA

4/16/2013

14

| 27

GSI Commerce’s Performance Best PracticesGSI Commerce’s Performance Best Practices

■ Performance matters!

■ Define Performance Targets

■ Test both client-side AND server-side

performance

– Cross-browser testing

– Add server-side testing

– Tie everything together

| 28

GSI Commerce’s Performance Best PracticesGSI Commerce’s Performance Best Practices

■ Automate!

■ Get Test and Development on the same page

■ Get Proactive

■ Benchmark your site against your competition

4/16/2013

15

| 29

QUESTIONS?QUESTIONS?