Monitoring web application response times^lj a hybrid approach for windows

Mark FriedmanDemand Technology Software

markf@demandtech.comhttp://computerperformancebydesign.com

Monitoring Web Application Response

Times in Windows

1

Why is this web app running slowly?

2

Main Topics

Building scalable web applicationsYSlow scalability model of web page compositionPage Load Time and Round tripsLimitations of the approach

W3C Nav/Timing APIGoogle Analytics & RUM

Progress report on an ETW-based approach to web application performance monitoring

plus, sprinkle in some case studies & best practices

Presentation Outline

3

Themes:The Value of Response Time measurements

Service Level reporting Application response time measurements correlate with measures of customer

productivity & satisfaction

Queuing models, decomposition & other analytic techniques

Obstacles: Measurement data missing from standard Windows ASP.NET counters, but it is available

from other sources Request-Response boundaries are blurred in many AJAX applications Understanding how to set good response time objectives

Since human beings are adaptable & “ Good” and “Bad” are often relative to the application context See “Engineering Time,” by Dr. Steve Seow

Web Application Performance

4

Themes:Web application programming models and fashion change faster than

tools can adaptAJAX

e.g., Auto-complete in Google Search achieved using client-side Javascript & Asysnchronous Http web service Requests

High Availability and Scalability using n-tiered architectures typically, a Presentation Layer, Business Objects layer, and a Data Access Layer

HTML5HTTP/2

Effective performance tools are usually one or two generations behind emerging technology e.g., http://webpagetest.org

Web Application Performance

5

Fundamental concept in software performance engineering (SPE)

namely, f(x), such that f(x) reliably predicts Response time.

Factors can be linear (m+n…)multiplicative (m * n)exponential (mn)

Scalability models

Assuming rendering time inside the web client is minimal,

Web Page Load Time = Render Time RoundTrips * RTT

where

RoundTrips =

6

YSlow scalability model

7

Why is my web app running slowly?

Part 1.

8

Case Study.

Developed using ASP.NET (Server-side controls)Multiple tiers: presentation/business objects/data layer

Uses the Model-View-Controller (MVC) pattern

Key elements of the web Pagedata-rich Charting component (.NET Chart, based on the Dundas component)Chart definition used to generate the PDB Query and the results are mapped

to a Chart instance Library of Chart templates Machine selection Date/Time selection

9

Why is this web app running slowly?

10

Example:

Ask YSlow

11

Why is this web app running slowly?

Based on the influential work of Steve Souders*originally at Yahoosince migrated to Google

Google Chrome extensionRule-based Influenced:

Chrome PageSpeed Insights IE Developer ToolsFiddlerGlimpseetc.

* High Performance Web Sites, O’Reilly Media, 200712

What is YSlow?

Optimize for Page Load Time Request.Start DOM.Complete

Make specific recommendations on how to improve Page Load Time for a specific web pageInventorying the Document Object Model (DOM) after the composition & rendering

the of web page: Calculate # of Http objects and their size

Does not attempt to actually measure Page Load Time, however.

Lead to a standardization effort to wire performance timing data to the DOM & create a consistent way to access it

Navigation Time, Performance Timing & a High Resolution Clock

13

What is YSlow?

14

Page Load Time

Web Browser Http Server

Http GET Request (Uri)

Compose&

Render

Http RESPONSE

Note: Http is a sessionless protocol

Measurement techniques

• sniff network packets• monitoring service to

submit synthetic Requests• Real User Measurements

• attach handlers to DOM window.unload and window.load events

15

YSlow Chrome extension

16

Chrome PageSpeed

Response message often contains embedded references to additional resources needed to render the Page

e.g., image files style sheets javascript files 17

Page Load Time

Web Browser Http Server

Http GET Request (Uri)

Compose & Render

Http RESPONSE

HTTP interacts with the underlying TCP/IP networking protocols HTTP Response messages > Ethernet packet size (~ 1500

bytes) require multiple IP packets With large cookies and a large number of parms, GET Request

messages can even exceed the Ethernet packet size 18

Page Load Time

Web Browser Http Server

Http GET Request (Uri)

Compose & Render

Http RESPONSE

YSlow scalability model:

assuming web client processing time is minimal, then

Render Time RoundTrips * RTT

RoundTrips =

19

Page Load Time

Ask YSlo

w

20

Why is this web app running slowly?

YSlow scalability model:Browser Render Time RoundTrips * RTT

Web Browser performs page composition using the Document Object Model, or DOMYSlow Rule: Make fewer HTTP requestsYSlow Rule: Improve cache effectivenessYSlow Rule: Reduce the number of DOM elementsYSlow Rule: Compress the objects the page does need to load

Tuning is a process that attempts to drive # RoundTrips 1

RoundTripTime 0 21

YSlow Scalability model

YSlow never actually measures RTT, but other related tools can

RTT may vary across RequestsObjects can be geographically dispersed

Local cluster, remote, cloude.g., referencing 3rd-party, advertising services

TCP adaptive window scaling and other network congestion avoidance strategies

Ignores variability in the execution time of client and server-side codee.g., sort() a large list of elements, ora hi-res visualization component that scales nonlinearly with the size of the result set

Compression recommendations can be at odds with good code maintainability practices

22

YSlow Scalability model complications

Web Browsers create multiple TCP sessions to download referenced objects in parallelYSlow Rule: take advantage of parallel sessions by loading scripts last

Accurate rendering of the DOM requires that downloading a Javascript file blocks parallel downloadsScript may add elements to the DOM dynamically, call other scripts or reference

additional resources Browser assumes DOM rendering can only resume after the Javascript code

executes

Page Load Time = Browser Render Time +Script execution Time + (RoundTrips * RTT)/Sessions

23

YSlow Scalability model complications

Despite the many complications, the YSlow scalability model has proved very influentialBrowser Page Load Time is an end-to-end measurement of service

time, which is apt to be correlated with customer satisfaction

see, for example, http://www.slideshare.net/Strangeloopnet/37-lessons-ive-learned-on-the-performance-front-lines

Inspired development of related tools to measure Page Load Time

24

YSlow Scalability model

The YSlow scalability model is useful, but it is not adequate for many web applications

25

Case study: YSlow scalability model assessment

Page Load time is a measure of end-to-end response timeNavigation Timing measurements decompose overall

response time into Network, Server, and Client (i.e., web browser) components

YSlow is silent: the scalability of server-side componentsthe diversity of web client hardware (PCs, tablets,

phones) & software (iOS, Android, Windows)the performance of the client’s network connection

e.g., Internet vs. Intranet connections

26

YSlow Scalability model assessment

Despite the many complications, the YSlow scalability model has proved very influential

Sparked a standardization effort so Javascript developers could access the PLT measurements reliably across Browsers

Creation of standard DOM performance objects that are accessible to Javascript

Finally, now that actual web application performance data is available in the web browser, how do I get that data back to my data center for optimization & capacity planning?

27

YSlow Scalability model

28Standard DOM NavTiming object

Prompt for unload redirect App

cache DNS TCP Request Response Processing load

unload DOMContentLoaded

navigationStartredirectStart

redirectEnd

fetchStartdomainLookupStart

domainLookupEnd

connectStartsecureConnectionStart

connectEndrequestStart

responseStartresponseEnd

loadEventStartloadEventEnd

unloadEventStartunloadEventEnd

domLoadingdomInteractive

domContentLoadedEventStartdomContentLoadedEventEnd

domComplete

Http Request/Response and Rendering eventsWeb Performance Working GroupPerformance Timeline, Navigation Timing,

and High Resolution Time specs Supported in IE, Chrome, WebKit, and FoxFire

Event timings can be used to calculate:

Network latency (from the standpoint of the web client) Page Render time (once the page components are

received from the server) Entire sequence from navigation to page load completion:

29

Page Load Time Measurements

Navigation Timing spec: https://dvcs.w3.org/hg/webperf/raw-file/tip/specs/NavigationTiming/Overview.html#processing-model

Calculate network latency: responseEnd - fetchStart

30

Page Load Time Measurements

Calculate Page Render Time:loadEventEnd - responseEnd

31

Page Load Time Measurements

Calculate entire sequence: loadEventEnd - navigationStart.

32

Page Load Time Measurements

see https://developer.mozilla.org/

33

performance.now High Resolution Clock

<html> <head> <script type="text/javascript"> // Add load event listener. window.addEventListener("load", loadTime, false);

function loadTime() { // Get current time. var now = window.performance.now(); // Calculate page load time. var page_load_time = now - performance.timing.navigationStart; // Write the load time to the F12 console. if (window.console) console.log(page_load_time); } </script> </head><body> <!- Main page body is here. --> </body> </html>

Now that actual web application performance data is available in the web browser, how do I get that data back from the web client?What about the volume of measurement data that requires processing

& analysis?

Google Analytics uses web beacons to send the response time data to Google’s data center for analysis and reporting

Yahoo Boomerang project can be used to send web beacons with the data back to your data center

34

NavTiming API

Architecture of an ASP.NET applicationWindows OSTCP/IP(Clustered) IIS front end Web ServersASP.NET.NET CLRADO.NET (data layer)

35

Why is my ASP.Net app running slowly?

Dynamic HTML is session-oriented behavior layered on top of the HTTP

Web applications require state Who you areWhere you are (mobile apps)

They preserve state:During a session

TCP protocol is session-orientedRequires a connectionCalculates RTT per connection (used in re-try logic)

Between sessions (using cookies, etc.)

36

ASP.NET Application Architecture

Server-side Request processingEvent-oriented programming model (Postback)HttpContext wrapper around the HTTP RequestPersistent State

ViewStateSession StateApplication and Page Cache objectsetc.

ASP.NET Application Performance

37

38

IIS Architecture

User

Kernel

HTTP

TCP

IP

Network Interface

HTTP Kernel Mode Driver

(http.sys)

HTTP Response Cache

(Physical Memory)

LSSAS

WindowsAuthentication

SSL

Inetinfo

IIS Administration

FTP

Metabase

SMTP

NNTP

SVCHOST

W3SVC

W3wpW3wp

Mscoree.dll

http

<code-behind>.dll

Default.aspx

Application Pool

W3wpW3wp

Mscoree.dll

<code-behind>.dll

Default.aspx

Application Pool

WAS

Cache

net.tcp net.tcp

http

HttpApplication Event Handlers (IHttpModule)

IIS 7.0 Integrated Mode (pipeline)

Event EventBeginRequest PreRequestHandlerExecuteAuthenticateRequest PostRequestHandlerExecutePostAuthenticateRequest ReleaseRequestStateAuthorizeRequest PostReleaseRequestStatePostAuthorizeRequest PostMapRequestHandlerResolveRequestCache PostMapRequestHandlerPostResolveRequestCache PostMapRequestHandlerMapRequestHandler UpdateRequestCachePostMapRequestHandler PostUpdateRequestCacheAcquireRequestState LogRequestPostAcquireRequestState EndRequest

39

Extends the ASP.NET event model to all Http RequestsHttpApplication Event Handlers (IHttpModule)

IIS 7.0 Integrated Mode (pipeline)

40

public class BoomerangBeacon : IHttpModule        {            public MyHttpModule()            {}            public void Dispose()            {}            public void Init(HttpApplication application)            {                application.BeginRequest += (new EventHandler(this.Application_BeginRequest));            }

}

w3wp.exe

Common Language Runtime (CLR)JIT compilerGarbage Collection threads

mscoree.dllmscorsvr.dllMyApp.dll

ASP.NET Application Architecture

41

Many, many Windows components are instrumented with ETWKernel objects include disk IO, process,

thread, paging, CPU TCP/IP, HttpServer, CLR

Providers may issue a current status Rundown

CallStacks can be captured

Event correlation:CPU ID, Process ID, Thread ID, File

Handle, Tcp Session ID (Port)42

ETW: Event Tracing for Windows

Many obstacles to using trace-based instrumentation effectively in performance investigationsSheer volume of trace data that can be generated (e.g., OS Dispatcher

ContextSwitch events) Cannot always be filtered effectivelyVery little documentation on specific events and how they are logically related

Inconsistent semantics (but only a few common patterns)Sequence: Begin, Step, EndStateChange(oldState, newState)Fork:JoinSend:Receiveetc.

43

ETW: Event Tracing for Windows

Resource MonitorWindows Performance tools (aka xperf) VS Profiler’s Concurrency Visualizer

inspired by Rico Mariani’s ETLStackBrowser programleveraged Vance Morrison’s TraceEvent .NET libray

and, introducing the Web Application Trace ExplorerGathers, analyzes, filters & reports on server-side events from TcpIP,

HttpServer, and instrumented application Scenarios, plus Boomerang Beacon data from web clients

Designed to help explore the semantics of these sparsely documented event streams 44

ETW tooling

IHttpModule to intercept the Boomerang beacon requests and turn them into ETW events

45

ETW tooling private void Application_BeginRequest(Object source, EventArgs e){

// Create HttpApplication and HttpContext objects to access         // request and response properties.         

HttpApplication application = (HttpApplication)source;HttpContext context = application.Context;

                string beaconUrl = GetBeaconUrl(); // Helper routine for

beacon url     string filePath = context.Request.FilePath;     string fileExtension = VirtualPathUtility.GetExtension(filePath);

if (fileExtension.Equals(".gif"))         {          if (filePath.Contains(beaconUrl))            {

... // Process the beacon parmsapplication.Response.End();

} }

}

Demo using sample data from webscorer.com

46

47

Main Navigation Window

48

49

HTTP activity by URL

50

Navigate to URL Detail

Boomerang beacon supportAdd the boomerang.js script to your web page HTML:

Initialize boomerang:

51

Web Application Trace Explorer

<script src="javascript\Boomerang\boomerang.js" type="text/javascript"></script><script src="javascript\Boomerang\plug-ins\navtiming.js" type="text/javascript"></script><script src="javascript\Boomerang\plug-ins\rt.js" type="text/javascript"></script

<script lang="javascript"> BOOMR.init({beacon_url: "boomerang.gif"});</script>

Boomerang beacon support

Add the BoomerangBeacon HttpModule from MeasurementWareWebServices.dll to web.config

52

Web Application Trace Explorer

  <system.webServer>     <modules>       <add name="MeasurementWareWebServices“              type="MeasurementWareWebServices.BoomerangBeacon"/>     </modules>   </system.webServer>

BoomerangBeacon classderives from IHttpModuleadds an EventHandler for HttpApplication.BeginRequest

In the BeginRequest event handler,intercepts the boomerang.gif GET Requestsparse the beacon parmsgenerate a MeasurementWareWeb ETW event whose payload includes

Page Load Time measurementsplus, IP Address and Tcp Port (unique session ID) of the Sender for correlation with

other HttpServer and TcpIP events

53

Web Application Trace Explorer

54

55

Beacon data summarized by URL

56

Navigate to URL Detail

57

Initial stage of the work to add Boomerang beacon data to the event collection and the displays is functionally complete

Prototype real-time collector that reports HTTP Response time measurements (by URL) as Windows performance counters is currently available

Contact me if your org is interested in participatingmarkf@demandtech.comSee my blog at

http://computerperformancebydesign.com for the latest status (until it reaches the Release to github stage)

Web App Trace Explorer: Project Status

?58

Questions