Best PracticesBest PracticesProtect your Online Revenue through Web Performance Testing

Melinda Carol-Ballou - Application Life-Cycle Management

Imad Mouline CTO Gomez

pp y g& IT Executive Strategies, IDC

Imad Mouline - CTO, Gomez

Optimizing Application Quality and Performance to Drive the BusinessOptimizing Application Quality and Performance to Drive the Business

Melinda BallouApplication Life-Cycle Management IT E ti St t iIT Executive Strategies September 2009

Copyright 2009 IDC. Reproduction is forbidden unless authorized. All rights reserved.

AgendaAgenda

Understanding the drivers and challenges for taking aUnderstanding the drivers and challenges for taking a life-cycle approach to application quality and performance

Coordinating quality and performance from requirements building through development and deploymentdeployment

Establishing process maturity and organizational strategies for successful application deployments

© 2009 IDC 3

Disruptive Trends Driving Automated Software Quality Adoption (ASQ)Disruptive Trends Driving Automated Software Quality Adoption (ASQ)• Evolving, flexible development paradigm with services creation will increasingly demand technology and business collaboration and drive for service quality within and outside the firewall strong agile emergenceservice quality within and outside the firewall – strong agile emergence

• Changing approaches to application testing as application packages evolve to new architectures

• End-user experience and business impact challenges of emerging web technologies (rich Internet), virtualization, cloud computing and Web 2.0

Global economic competition plus local compliance initiatives across• Global economic competition plus local compliance initiatives across geographies demand quality, adaptability and rigor

• Complex sourcing/off-shoring necessitate strong teaming and effective testing and quality management; limited infrastructure creates barriers

• License and delivery mechanisms with SaaS for new models, choices, customer support, speed adoption, limit complexity (human and systems)

© 2009 IDC 4

pp , p p , p y ( y )

“Quality Gap”: High Cost of Failure “Quality Gap”: High Cost of Failure

Poor Quality = Increased Business Risky

($$$$$)Lost Revenue

($$$$$)Lost Customers Increased CostsLost Revenue Lost Customers Increased Costs

L t P d ti it L P fitDamaged Brand

© 2009 IDC 5

Lost Productivity Lower ProfitsDamaged Brand

Two Goals of Effective IT/Business AlignmentTwo Goals of Effective IT/Business Alignment

Innovation:Innovation:Maximize Upside

Through Technology-Enabled Business

New Business Value

Processes

ReducedExposure

Compliance:Compliance:Minimize Downside

Through Risk Management

© 2009 IDC 6

g

Key IT Challenges: Business-Criticality

Key IT Challenges: Business-Criticality

Today’s applications are high-visibility, and carry a high cost of failure:high cost-of-failure:

Customer “self-serve”

Supplier/channel integrationSupplier/channel integration

Internal applications that automate critical business processes

“Network effect”: failure in one applicationcan mean loss of service in others.

“Outside-in” Testing is Increasingly Important Due to Web Application Delivery Chain Complexity

© 2009 IDC 7

Performance Problems: a Leading Cause of Failure for New ApplicationsPerformance Problems: a Leading Cause of Failure for New Applications

Q. When your software fails to satisfy users, the top 3 reasons are:

Flawed specification

Poor performance /capacity/response time

Under-estimated resources /time required

p

Inability to integrate with other software

Failure of app. to evolve with business needs

Flawed implementation of a correct spec

Data quality issues

Dissatisfaction with user interface

Inefficient or incorrect configuration

Quality of code /lack of testing

Flawed implementation of a correct spec

© 2009 IDC 8

0 5 10 15 20 25 30 35(% of respondents)Source: IDC n=467

IT Business Challenges: Silos, Gaps IT Business Challenges: Silos, Gaps

The need for a quality life-cycle is key since G2000 organizations are split across interdependent groups:

Business/users stakeholdersDevelopersQA professionalsOperational staff

Increased complexity for both technology and source (offshore, etc.) drives need for coordination

Must extend the Quality life-cycle across phases andMust extend the Quality life cycle across phases and groups

© 2009 IDC 9

Rise of Global Teams and BusinessesRise of Global Teams and Businesses

EuropeanUI Team EuropeanBusiness Unit

APACBusiness UnitDev/QA/

Dev/QA/Support

US Business Sponsors

Content

SupportContent

Dev/QA/Support

BusinessAnalysts

© 2009 IDC 10

Coordinating across the Life-CycleCoordinating across the Life-Cycle

• Coordinating requirements, testing, and operational performance are key across core emerging technologies

• Slow response times for key business areas are problematic

• Organizations should target quality life-cycle approaches th h i t it t t t i t ti dthrough requirements, unit test, system integration and pre-deployment and performance testing and change management

• Leveraging test automation for synthetic transactionLeveraging test automation for synthetic transaction monitoring enables users to manage user experience with applications

• As business requirements change, a cogent life-cycle approach enables adaptive testing and monitoring for flexible responses

© 2009 IDC 11

Establishing Process and Org StrategiesEstablishing Process and Org Strategies

Engage Upfront Business input with requirements and QA and change activities via organizational and process strategy

“Customer advocates” – attuned to business and IT– Design a test plan based on how real users interact with

your applications• Where they do it

– LA, Edmonton, London, Bangkok, NY, etc• What they do

– Key tasks, transactions, and business processes• How they do it

– Browsers galore, mobile devices• When they do it (peak times, spikes and how frequently)

Better estimate effort better coverageEarly requirements/test/design helps reveal design problems –

© 2009 IDC 12

y q g p g pmanaged change key

Establishing Process and Org StrategiesEstablishing Process and Org Strategies

Involve LoB & Ops in projects earlier

Refine test plansRefine test plans– Identify key geographies and third-party components to test– Help define third-party SLAs, and overall “business SLAs”

Coordinate testing of external components and services delivered from outside the datacenter

Anticipate deployment and configuration issuesAnticipate deployment and configuration issues Leverage tools, tests and skill sets across the life-cycle

to ensure functionality, availability, and performance y y pgoals are met

© 2009 IDC 13

Application Quality – Design and DeliveryApplication Quality – Design and Delivery

Quality Evaluation Matrix

•Easy to Use

U di t bl

•Easy to Use

B h

esig

n

•Unpredictable Behavior

•Behaves as Designed Quality of

Customer ExperienceD •Difficult to Use

•Unpredictable Beha ior

•Difficult to Use

•Behaves as D i d

Experience

Behavior Designed

© 2009 IDC 14

Delivery

App Deployment & Support: Old ViewApp Deployment & Support: Old View

Define

DEV

SLAs, Provision

Test/Tune Monitor Support

Issues

Result:Little input into specifications or developmentLittle input into specifications or developmentLittle leverage between development/ops of testing/monitoring investmentsTesting/tuning LATE in the cycle!

© 2009 IDC 15

Testing/tuning LATE in the cycle!

Closing the Loop: Leverage Skills & Tools for Agile, Iterative ApproachClosing the Loop: Leverage Skills & Tools for Agile, Iterative Approach

•Shared goals,

DesignDefine

g ,•Shared metrics,•Share tests, tools and skills Define

Develop

tools,and skills•Shared systems

Support

Deploy & Monitor

Test & Tune

© 2009 IDC 16

Establish a Common LanguageEstablish a Common Language

“Network statistics look great.”“All servers are available ”

“Sales says customers can’t

“Call Center’s swamped with All servers are available.

“All databases are available.”“99% of our monitored apps are OK”

“Service levels are within limits.”

customers can t find product in the

database.”

swamped with Order Status

inquiries.”

“We’re not “Order Service levels are within limits.meeting our first call resolution metric today.”

volume is down.”

Business

Bridge the communication gap!IT

© 2009 IDC 17

Bridge the communication gap! Establish a common language!

Bridging the Customer Experience GapBridging the Customer Experience Gap

Business M

“Is the end-to-end business processManagement

Customer End-to-End Usage Scenarios

business process working?”

Customer ExperienceManagement

g

Web Voice Web Services email FAX

“Can customers do what they want to do?”

Application

Web, Voice, Web Services, email, FAX…

“Can I proactively fix things before

want to do?

Server, Network & Element Management

Management fix things before they impact the business?”

© 2009 IDC 18

Element Management

Closing a ‘Customer Experience’ GapClosing a ‘Customer Experience’ Gap

1. Establish a common language between IT and the business, designate a liaison

2. Agree on business process measurement priorities

3. Implement testing/management for end-to-end p g gbusiness processes – from the customer/end-userperspective

4. Eliminate silos and close the loop between development and operations

S f5. Share goals, metrics, feedback with the entire team: make Customer Experience Management part of the culture

© 2009 IDC 19

SummarySummary

•Coordinate a Quality Life-Cycle approach from requirements through to operations to enable adaptive businesses

•Schisms between business, development, testers and operations must be addressed

•IT groups and the business must build a common language, common metrics, and common tools and practices

•Focusing on the customer experience will help IT achieve this along with core incentives to collaboration

•Making “customer experience” the culture is key to help bridge the gap and to enable successful collaboration across groups

© 2009 IDC 20

Best Practices: Protecting your Online Revenue

BP #1: Establish a Common Languagebetween IT and the business

BP #2: Make Customer Experience Management part of the culture

BP #3: Implement testing & management for end‐to‐end business processes – from the customer/end‐processes  from the customer/enduser perspective

BP #1: Establish a Common Language Between IT & the Business:Customer Satisfaction Index 

Apdex (Application P f I d )

Quickly isolate and prioritize i i f fPerformance Index)

Single measurement of user satisfaction with W b f

opportunities for performance improvement based on real customer satisfaction

h fWeb performance  Based on response time  thresholds

Why is customer satisfaction decreasing? What do users have in 

?common? How do I compare against my competitors?Satisfied Tolerating Frustrated

http://www.apdex.org 2007

Customer Satisfaction Index: An Example

A l d i i f i dAs load increases customer satisfaction decreases

Customer satisfactionsatisfaction decreases

Load increases

BP #2: Make Customer Experience Management Part of your Culture ‐ Design your Web site for a Multi‐browser World

ill i k ll b ?Will it work across all browsers?Functions correctlyFunctions correctly

Does not function correctly = brand damageDoes not function correctly = brand damage

Make Customer Experience Management Part of your Culture ‐ Test & Monitor your Key Transactions

Further analysis shows sequence 0 of the transaction - the login page - is

However sequence 1, the authentication step, is i t d b junaffected impacted by a major performance issue

Make Customer Experience Management Part of your Culture – Ensure that your Applications Will Scale Under Load

Obama family exclusively wears leading fashion brand for inauguration ceremony

Leading fashion brand’s web application crashes leading to short and long term loss of revenue and brand equity

BP #3: Implement Testing/Management  for End‐to‐end Business Process from the Customer/End‐user Perspective 

Behind‐the‐firewall testing 

tools: “OK”

…user is NOT happy

Load Balancing

Users

The traditional view of Web Application Delivery

Web Servers

App Servers The traditional view of Web Application Delivery

Storage

DB Servers

Mobile Components

Traditional zone of control

Critical to Test Across Entire Web Application Delivery Chain

The Web Application Delivery Chain

Behind‐the‐firewall testing 

tools: “OK”

…user is NOT happy

Local ISPLoad Balancing

Users3rd Party/

Cloud ServicesBrowsers 

and Devices

• Inconsistent geo performance• Bad performance under load

• Network peering problems

• Poorly performing 

Web Servers

App Servers • Network peering 

problems

Bad performance under load• Blocking content delivery• Incorrect geo‐targeted content

p• Bandwidth throttling• Inconsistent connectivity

• Configuration errors

p gJavaScript

• Browser/device incompatibility

• Page size too bigT

Internet

MajorISP

Storage

DB Servers

problems• Outages

• Configuration issues• Oversubscribed POP

• Application design issues

• Code defects• Insufficient infrastructure

• Too many objects

• Low cache hit rate

• Network resource shortage 

• Faulty content transcoding SMS i /

Content DeliveryNetworks

Mobile Carrier

Mobile Components

• Poor routing optimization• Low cache hit rate

• SMS routing / latency issues 

Zone of customer expectationZone of customer expectationTraditional zone of control

Load Testing 2.0Performance Testing For A Web 2.0 World

Self‐service testing gives you control to test as needed

T t i t l & t l bTest internal & external web application components to ensure transactions perform under load

Test cloud‐based applications &Test cloud‐based applications & services

SaaS based Load Testing 2.0 solutions enable on‐demand, ,scalable testing without associated hardware investment & maintenance costs

C bi hi h l Test the full web application delivery chain all the way to your end users’ desktops

Combine high volume application load testing with broad geographic experience testing from the outside-in

Backbone

Gomez Reality Load XF: On‐Demand Realistic Load Testing from Browser to Data Center

Last MileBackboneHigh volume load (HTTP, Browser)

Find infrastructure breaking pointsDefine capacity headroom

Last MileReal‐world load

Find user experience breaking pointsAccurately measure response time

Local ISPLoad Balancing

Users3rd Party/

Cloud ServicesBrowsers 

and Devices100,000+ consumer‐

d

100,000+ consumer‐

d

100,000+ consumer‐

dWeb Servers

App Servers

grade desktops

168+ countries

grade desktops

168+ countries

grade desktops

168+ countries100+ 

commercial‐100+ commercial‐100+ commercial‐

Internet

MajorISP

Storage

DB Servers

2,500+ ISPs

Major mobile 

2,500+ ISPs

Major mobile 

2,500+ ISPs

Major mobile 

grade nodes & data centers

grade nodes & data centers

grade nodes & data centers

Content DeliveryNetworks

Mobile Carrier

Mobile Components

carriers around the globe

carriers around the globe

carriers around the globe

Ensuring Performance of All 3rd Party Components

Company: Online Retailer• Several 3rd Parties now involved in serving up key content

G l t lid t f f ti li ti• Goal was to validate performance of entire application

3rd Party/ Browsers Local ISP

y/Cloud Services and devices

Load Balancing

Web 

Load Balancing

Web 

Users*

Internet

Servers

App Servers

DB 

Servers

App Servers

DB MajorISP

Storage

Servers

Mobile Components

Storage

Servers

Mobile Components

Content DeliveryNetworks

Mobile Carrier

ComponentsComponents

Response Times Rise Due To 3rd Party Object Error

The load increases throughout the test

The transaction rate increases and then falls off as responsefalls off as response times climb

Errors are seen, all on a 3rd party objecton a 3rd party object

3 d t h d i ffi i t f ll d d li ti• 3rd party hardware was insufficient for overall demands on application• Based on SLAs 3rd party had to improve performance to get paid

Ensuring Performance in Key Markets

Company: Regional Online News Source• Began testing for the 2008 election season

G l t lid t ll f f i i 2 k i• Goal was to validate overall performance focusing in 2 key regions

3rd Party/ Browsers Local ISP

y/Cloud Services and devices

Load Balancing

Web 

Load Balancing

Web 

Users

*

Internet

Servers

App Servers

DB 

Servers

App Servers

DB MajorISP

Storage

Servers

Mobile Components

Storage

Servers

Mobile Components

Content DeliveryNetworks

Mobile Carrier

ComponentsComponents

No Performance Issues Detected From Data-Center

There was only 1 page error and

Increase and hold load and not exceed response times of 4 seconds and Success Rate of 99%

Page response times stayed under

11 errors total out of 60000+ transactions

times stayed under 4 seconds, outside of one brief blip

By traditional test standards the test passed

Performance Issues Detected From Real User Desktops

Key geographies for this customer are New York and PennsylvaniaNew York and Pennsylvania

Last Mile data showing substantial number ofLast Mile data showing substantial number of measurements greater than 4 seconds

The Gomez Platform: Web Application Experience Management

SelfSelf‐‐Service SaaS PortalService SaaS Portal

20• Compatibility• Compatibility

Web Cross‐Browser 

Testing

Web Cross‐Browser 

Testing

• High‐volume• High‐volume

Web Load and Performance 

Testing

Web Load and Performance 

Testing

• Real‐user monitoring• Real‐user monitoring

Web Performance Management

Web Performance Management

• Web performance• Web performance

Web Performance

Business Analysis

Web Performance

Business Analysis

20Compatibility• Functionality• Performance

Compatibility• Functionality• Performance

High volume backbone load

• Real‐world Last Mile load

High volume backbone load

• Real‐world Last Mile load

Real user monitoring• Last Mile monitoring• Backbone monitoring

Real user monitoring• Last Mile monitoring• Backbone monitoring

Web performance analytics

• Benchmarks• Business dashboard

Web performance analytics

• Benchmarks• Business dashboard

Web 2 0 RIA Streaming Mobile

Dashboards, Metrics & Analytics

Dashboards, Metrics & Analytics

Recording & ProvisioningRecording & Provisioning

Education &Best PracticesEducation &Best Practices

APIs &Data Feeds APIs &

Data Feeds Alerting & DiagnosticsAlerting & Diagnostics

Web 2.0, RIA, Streaming, Mobile 

World’s Most Comprehensive Testing NetworkWorld’s Most Comprehensive Testing Network168+ countries    2,500+ ISPs168+ countries    2,500+ ISPs

100+ commercial nodes and data centers

100+ commercial nodes and data centers

500+ combos of 

browsers & O/S

500+ combos of 

browsers & O/S

5,000+ supported 

mobile devices

5,000+ supported 

mobile devices

100,000+ consumer‐grade 

desktops

100,000+ consumer‐grade 

desktops

Gomez Customers Enjoy Measurable Benefits

Increased conversions 10% 

Reduced homepage load time from 11 3 d t 3 4 d11.3 seconds to 3.4 seconds

Improved page load times 23%

Reduced seven‐step transaction time by 50%

Saved 50%+ in staff and fees

Reduced downtime 45% 

y

Achieved under 3 second

Validated decision to consolidate

Achieved under 3 second response time and 99%+ availability

Validated decision to consolidate three data centers

Appendix

Ensuring Performance Of Login Process

Company: Online presence for a popular TV show

• Following episodes of the TV show the web site sees high traffic spikes

• Goal was to achieve 1500 logins per minute• Goal was to achieve 1500 logins per minute

• Load tested DB to improve performance in anticipation of another traffic spike

3rd Party/ Browsers 

1Local ISP

y/Cloud Services and devices

Load Balancing

Web 

Load Balancing

Web 

Users

Internet

Servers

App Servers

DB 

Servers

App Servers

DB MajorISP

Storage

Servers

Mobile Components

Storage

Servers

Mobile Components

Content DeliveryNetworks

Mobile Carrier

ComponentsComponents

Application Bottleneck Causes Immediate Response Time Issue

• As users were added the• As users were added, the response time of step 3 (the login) climbed immediately

• The test bottlenecked at 160 logins per minute (Goal 1500)

• But quickly dropped off as users received server errors

• New login query was not optimized and was bottlenecking the database

’ CPUservers’ CPUs