Developer and Fusion Middleware 1 _ Christian Ledwidge _ Application Performance Monitoring - Improve User Experience reduce user abandonment.pdf

The most comprehensive Oracle applications & technology content under one roof The most comprehensive Oracle applications & technology content under one roof

Applica'on Performance Monitoring: Improve user experience, reduce user abandonment

Chris&an Ledwidge Director Solu&ons Architecture Monitoring Quest So9ware

The most comprehensive Oracle applications & technology content under one roof

Going to tell you a li<le story before we begin

•  Established well established travel company wants to get into the online travel booking engine business –  They have an established brand in the package holiday market with

their own fleet of aircra9 to service –  Margins are under the gun and they need to find new areas for

revenue genera&on •  Its seems an obvious choice they know the travel business a9er all •  Have a good name that should help drive users to their new website •  They ramp up a big team of marke&ng, IT, and business owners to kick this

project off •  Some solid ini&al success and word of mouth spreads and site hit counts

move in the direc&on they had hoped for


Story con&nued •  BUT! •  Booking numbers are missing forecasts by a solid margin •  Calls to their call center are going thru the roof

–  Business: these call are ea&ng up our margin aren't they?

•  Nobody in IT seems to be able to tell the business why they are not conver&ng the solid site hit rate into completed transac&ons

•  Lots of finger poin&ng and conspiracy theories are geSng thrown around –  It’s the network! –  No it’s the DB! –  Its that new Java applica&on that THEY wrote! –  Why did we ever go Virtual?

•  Only thing the business cares about it why are they not conver&ng all of these poten&al customers into paid clients?


Part One

User Experience Management (UEM) Indicators

•  What are they and what do they do

•  Five key indicators and how they can be used

•  What sources can be used to obtain indicators

•  Important points to remember


UEM Indicators: Bridge the Gap between Business and IT

Business

Useable? Stable? Right Content? Right Features? Fast Enough?

Available? Fast Enough? Short MTTR? Proactive Problem Solving?

IT

Back-end Systems Web User Web App

UEM Indicators • User counts • Changes in user behavior • Geographical variations • Problem impact • Service level agreements


UEM Indicator: User Counts

Business IT

Use Case Twenty-five users have registered in the last 15

minutes, but normally at this time, I see 200 or more people registering.

Is the reduc&on in registra&ons isolated to a specific region or is it across the board?

Are there any outages impac&ng registra&ons that we need to communicate to the business?

How many users are being impacted and from which region(s)?

Did our promo&onal campaign stop producing results or is the link broken?


UEM Indicator: Changes in User Behavior

Business IT

Use Case Typically 25% of users complete this transaction, but today we see fewer than 15%; why the changes in

user behavior?

Did the new site rollout hide the submit bu<on to where some users can find it anymore?

Where along the path are we losing the users and when did it change?

Are problems with the applica&on causing problems with this transac&on to where users can’t complete it?

How can we help the business restore service and measure the business impact of the change?


UEM Indicator: Geographical Varia&ons

Business IT

Use Case We see unexpected geographical variations in user

volumes from our EMEA region.

How many users are being impacted and for how long?

Is this a problem we are causing or is it one of our applica&on delivery partners?

Is our European proxy server having performance issues?

Is there a specific service provider that we or an ISP that our users are using experiencing issues?


UEM Indicator: Problem Impact

Business IT

Use Case A user just called in and reported a problem that our monitors didn’t pick up; what is the problem impact?

When did the problem start and what area of the site is being impacted?

Is the error being displayed to end users or is it just an image failing that they don’t really see?

Is there a specific web server throwing these excep&ons?

Is the right web administrator involved to get this fixed or diagnosed?


UEM Indicator: Service Level Agreements

Do the performance agreements that are in place guarantee a sa&sfactory user experience?

Are the IT systems mee&ng the agreed-‐upon delivery of services to end users?

Business IT

Use Case SLA goals are not being met?

When the SLA viola&ons started occurring were there devia&ons in transac&on conversions?

Were our Web, App, or Database systems at fault or was it an external interface?


User Experience Management Data Sources

Data Sources

User Experience Management Indicators

•  Web Log Files •  Synthetic Transaction Robots •  Client-Side Tagging •  Web Traffic Analysis Appliances •  Content Capture Appliances •  User Surveys

•  User Counts •  Changes in User Behavior •  Geographical Variations •  Problem Impact •  Service Level Agreements


Data Source

Usefulness •  Error tracking •  User click stream analysis •  Demographic analysis •  Performance

Limitations •  Difficult to identify

individual user sessions. •  Data set is partial due to

caching.

Web Log Files


Usefulness •  Measures transaction

performance and availability via scripted transactions executed from remote locations at set intervals.

•  Sometimes referred to as a “clean room” measurement.

•  Good for service level reporting.

Limitations •  Measures only a small

percentage of locations. •  Exercises a small

percentage of the technology stack in larger environments.

Synthetic Transaction Robots Data Source


Usefulness

•  Good for measuring true client performance

•  Gathering browser-based events needed to understand Web 2.0 user interactions

Limitations •  Inserting and managing

the scripts adds complexity and overhead to the application

•  Data security concerns

Client-Side Tagging Data Source


Usefulness

•  100% sampling of transaction performance, volume, and errors.

•  Analysis has many perspectives including by server, by geography, by ISP, and even by user.

Limitations •  Asynchronous

applications make it difficult to see cause and effect.

•  Analyzed data set is partial due to external caching.

Web Traffic Analysis Appliances Data Source


Usefulness

•  Stores the full user sessions, including the data returned from the web server.

•  Session replay and other perspectives on the session that are attainable only when full payloads are accessible

Limitations

•  Storage capacity concerns

•  Privacy and security concerns

Content Capture Appliances Data Source


Usefulness

•  True voice-of-the-customer

•  Can validate metric-based hypotheses

•  Answer questions like: o  Why are customers

happy? o  Why are customers

ready to leave?

Limitations

•  Typically only small samples are gathered, and small samples may represent extremes.

User Surveys Data Source


Part Two

•  Measuring user experience in web 1.0 and 2.0 environments

•  Top 5 performance measurement challenges

•  Applying approaches to address the challenges

•  Key elements to remember


User Experience Management Performance Measurements

•  Web 1.0 – Simple requests and responses

•  Web 2.0 – Asynchronous requests and responses

•  CDNs and content hos&ng – third-‐party effects

Processing Time Network Time Client Time

Request

Response

Back-end Systems Web App

Internet


Web 1.0 – Simple Page Request No Third-‐party

Client Server Client Server

1 2

Connection 1 Connection 3

GET main.html

GET image1.gif

GET frame1.html

Client Server

4

3

Connection 2

GET style.css

GET news.asp

1.  Request ini&ated

2.  Server gets first packet

3.  Server sends last packet

4.  Final response received by client


Web 2.0 – Asynchronous Requests

Client Server

1

2

4

3

Connection 1

GET item.js

1.  Request ini&ated

2.  Server gets first packet

3.  Server sends last packet

4.  Final response received by client


CDNs and Remote Content – Third-‐Party Effects

Client Server Client Server Client Akamai Client Akamai

2

4

3

Connection 1 Connection 2 Connection 3 Connection 4

GET main.html

GET perf.js GET frame1.html

GET style.css

GET news.asp

1


User Experience Management Measuring Performance to Solve IT Challenges

IT

Web User Back-end Systems Web App

“We learn about most problems only when a user

calls in”

“The user tries to explain her problem, but it really doesn’t

help me resolve it.” “We initially can’t tell if the

issue is on the client, in the network, or buried in the backend infrastructure.”

“We don’t know how many and which users are having

this experience.”

“We lack historical data to proactively detect unwanted performance trends and their

root cause.”

Internet


Challenge One “We learn about most problems when users call”

Management Approach Minimal Coverage BeCer Coverage Best Coverage

•  Log parser-‐-‐ alerts on errors or performance

•  Log parser-‐-‐ alerts on errors or performance

•  Synthe&c package-‐-‐to add transac&on perspec&ve

•  Synthe&c package-‐-‐to add transac&on perspec&ve

•  Selec&ve page instrumenta&on-‐-‐captures third party content provider issues

•  Passive sniffer-‐-‐low overhead capture all solu&on


Challenge Two “User explana'on is not useful for resolu'on”


•  Call the user(s) back and inquire about issue

•  Enlist support teams to recreate issue

•  Selec&ve page instrumenta&on-‐-‐captures desktop events

•  Scrape web logs for traces of informa&on about the issue

• Use online desktop sharing to record customer scenario

•  Passive sniffer-‐-‐captures full content and recreates scenario


Challenge Three “Can’t isolate issue to client, network, or backend”


•  Look for anomalies in web logs-‐-‐dismiss problem if no anomalies

•  Look for anomalies in web logs-‐-‐dismiss problem if no anomalies

•  Selec&ve page instrumenta&on-‐-‐captures third-‐party content performance

• Use synthe&c solu&on to isolate issue-‐-‐if it shows up

•  Passive sniffer-‐-‐differen&ates client from network &me for requests


Challenge Four “Don’t know how many users experiencing issue”


•  Count anomalies in web logs-‐-‐es&ma&ng impact from counts

•  Count anomalies in web logs-‐-‐es&ma&ng impact from counts

•  Selec&ve page instrumenta&on when third-‐party content being served

•  Use synthe&c solu&on to judge impact according to bad loca&ons

•  Passive sniffer-‐-‐assess impact of all first-‐party requests


Challenge Five “No historic info to detect performance anomalies”


•  Customer sa&sfac&on surveys

•  Web log analysis tools with long term reten&on

•  Selec&ve page instrumenta&on when third-‐party content being served

•  Web log analysis tools with long term reten&on

•  Synthe&c solu&on with base lining capabili&es

•  Passive sniffer-‐-‐for first party base-‐lining and anomaly detec&on


Important Points to Remember

•  User Experience Management encourages a common understanding between IT and the business

•  Common indicators enable collaboration

•  Sources of User Experience Management information are many—use those that best fit your needs


Part Three •  Why Manage Transac&on Conversions

•  Web Analy&cs vs. Transac&on Conversion Analysis

•  Measuring Transac&on Conversions

•  Top 5 Challenges to Managing Conversions

•  Points to Remember


Why Manage Transac&on Conversions

•  Why do we manage transac&ons the way we do?

•  Example Transac&on Types

ü Shopping Carts

ü Ad campaigns

ü Customer registra&on

ü Login


Complementary solutions

Why Transac&on Analysis—We already have Web AnalyCcs? Web Analy'cs •  Primary goal: How effec&ve your marke&ng is…

•  Answers “How effec&ve is my marke&ng?”

•  Observe how users use your website:

– What transac&ons they use and how o9en

–  Click stream analysis

Transac'on Analysis •  Primary goal: What kind of experience your users have…

•  Answers “What influenced user transac&on conversions?”

•  Observe how user transac&on conversions are affected by:

–  Content

– Applica&on func&onality

– Applica&on useability

– Applica&on stability


User Experience Management TransacCon Conversion Analysis


Measuring Transac&on Conversions

Start

Transaction

Step 2 End Step 3

Hit Hit Async Hit Hit Async Hit Hit Async Hit Hit Async

Page 1 Page 2 Page 3 Page 4

Session

Capturing Measurements

•  Unique Request ID (Off the wire) –  URL, Page Title, Content, Variable, Loca&on,

etc.., –  Persists current transac&on state per session in

memory

•  JavaScript Instrumenta&on –  Captures page or request level

informa&on per session

–  Reports transac&on progress back to central monitoring server

–  Result sent to a third or first party


User Experience Management Increasing Conversion Rates through Conversion Analysis

“How much money have we lost because of this

problem?”

“We know where people are abandoning the website but we don’t

understand why.”

“I need to understand bad conversion trends immediately to reduce lost business.”

“We can’t identify how many and which users are associated with conversion issues.”

“When people abandon the site, they often call customer service – which drives up our cost of service.” Business


“Revenue loss -‐ a Key Indicator example”

My Transac'on Record

Transac&on Name Buy Funnel

Username Mike Smith

Browser IE 8

Cart Amount $625

Transac&on State Ac&ve

Sale Amount N/A

Lost Sale Amount N/A

1.  Recognized add-‐to-‐cart page 2.  Page analyzed

•  Parse $$ amount 3.  Updates Transac&on Record

•  Add to current Cart Amount 4.  Transac&on Completes Successfully 5.  Increment Sale Amount

•  Set Lost Sale Amount to 0



Username Mike Smith

Browser IE 8

Cart Amount ….

Transac&on State Complete Success

Sale Amount $625

Lost Sale Amount 0


“Understanding Bad Conversion Trends”

•  Quan&ta&ve measures available in Analy&cs and UEM •  Allows us to form a hypothesis about conversion trends •  Qualita&ve valida&on is the next step to validate •  Session replay is that valida&on



Username Mike Smith

Browser IE 8

Cart Amount $625


Sale Amount N/A



“The 5 W’s of Transac&on Abandonment”

•  Where are users dropping? •  Which users are dropping? •  Why are users dropping? •  When did users start dropping? •  What were users doing when they dropped?

My Transac'on Record Transac&on Name Buy Funnel

Username Mike Smith

Browser IE 8

Cart Amount $625


Sale Amount N/A



Username Mike Smith

Browser IE 8

Cart Amount $625


Sale Amount N/A



Username Mike Smith

Browser IE 8

Cart Amount $625


Sale Amount N/A



Username Mike Smith

Browser IE 8

Cart Amount $625


Sale Amount N/A



Username Mike Smith

Browser IE 8

Cart Amount $625


Sale Amount N/A



Username Mike Smith

Browser IE 8

Cart Amount $625


Sale Amount N/A



“People calling customer service drive up costs”

•  Relying on customers to call in when problems with transac&ons is costly –  Lost revenue –  Damage to brand—reduces repeat purchases

–  Call center personnel are expensive

•  Transac&on Conversion Analysis provides insight to: –  Spot unwanted conversion trends before

they impact the business –  Increases efficiency of call-‐center personnel

—look at a user session to recreate the problem


Points to Remember

ü Transaction analysis is insurance for achieving and maintaining desired business outcomes

ü Weigh the costs of not doing versus doing conversion analysis

ü Is analytics enough or do you need something more to make you analytics actionable?


Thank you for your &me

Please drop by the Quest booth in the vendor hall if you would like to have a look at our approach to Applica&on Performance Management

Documents

Developer and Fusion Middleware 1 _ Christian Ledwidge _ Application Performance Monitoring - Improve User Experience reduce user abandonment.pdf