Performance Metrics in a Day with Selenium

Performance Metrics In A Day

Shane Hender

Mark Watson

Agenda

Goal is to show how you can enhance your automation tests to gather performance metrics

Keep the short presentation as practical as possible with enough code snippets to give you a starting point

One more thing…

HTTP Archive (HAR)

What? UTF8 Text based JSON encoding to represent

performance data

Why? Many libraries to convert from JSON to objects More tools allowing import/export/view HAR Becoming the standard format

Snippet of HAR format

{ "log": { "version": "1.1", "creator": { "name": "Firebug", "version": "1.7.0" }, "browser": { "name": "Firefox", "version": "4.0" }, "pages": [ { "startedDateTime": "2011-03-31T16:56:50.455-07:00", "id": "page_62901", "title": "Google", "pageTimings": { "onContentLoad": 431, "onLoad": 3148 } } ],

"entries": [ { "pageref": "page_62901", "startedDateTime": "2011-03-31T16:56:50.455-07:00", "time": 250, "request": { "method": "GET", "url": "http://www.google.com/", "httpVersion": "HTTP/1.1", "cookies": [ { "name": "PREF", "value": "ID" },

HAR Viewers http://www.softwareishard.com/har/viewer/

Fiddler UI can import HAR files ShowSlow webapp can be used to archive and view

HAR files

Which Metrics? Overall page load time DOM loading/interactive/complete, browser 1st

render, … Per-item timings

DNS, SSL connect, time to first byte, total time to download bytes, …

Headers, status codes, and content For a more detailed picture of the reasons for the

page performance Can help with debugging performance, e.g. were

items compressed, or not being loaded dynamically, or certain items not being prioritized for above-the-fold rendering

Methods for gathering metrics Setting your own timings in test code Using the new ‘Navigation.Timings’ or Web

Timings standard built into browsers Using browser plugins that report back the

timings to you, e.g. WebPageTest Per HTTP request logs via Selenium’s built-in

proxy Routing the browser traffic through a local

proxy and gathering the statistics from there. Network traffic capture

Method 1: Own Timings

WebDriver driver = new ChromeDriver();Date start = new Date();driver.get("http://www.webmetrics.com");Date end = new Date();

Method 2: Web Timings Currently Chrome and IE9 supported, coming soon

for Firefox http://w3c-test.org/webperf/specs/NavigationTiming/

WebDriver driver = new ChromeDriver();// A "base url", used by selenium to resolve relative URLs String baseUrl = "http://www.webmetrics.com"; driver.get(baseUrl); JavascriptExecutor js = (JavascriptExecutor)driver; Long loadEventEnd = (Long) js.executeScript("return

window.performance.timing.loadEventEnd");Long navigationStart= (Long) js.executeScript("return window.performance.timing.navigationStart");Long answerToAllProblems = loadEventEnd - navigationStart;

Method 2: Web Timings Could also modify this by using your own

JavaScript loaded into the page to set timings

Also would work in combination with the already set Web Timings E.g. Set a Date().getTime() variable when some dynamically

loaded content is loaded into the DOM

Long perceivedLoadTime = (Long) js.executeScript("return elementInsertTime – window.performance.timing.navigationStart");

Unfortunately it doesn't give timings per item downloaded, e.g. images, css, js, ....

Method 3: Browser Plugins Typically can give compute time metrics

Javascript execution time CPU usage Render times

IE8 - WebPageTest CPU usage Video capture

Firefox - Firebug Net Panel + NetExport https://github.com/lightbody/browsermob-page-perf https://github.com/AutomatedTester/AutomatedTester.PagePerf.git

DynaTrace browser plugin (FireFox/IE on Windows) Good breakdown of stats (Javascript execution times, CSS times,

render, 'first impression' time). http://ajax.dynatrace.com/ajax/en/

Chrome – some export from the Developer Tools Network Panel?

FirefoxProfile p = new FirefoxProfile();

try { p.addExtension(new File("c:/firebug-1.6.0.xpi")); p.addExtension(new File("c:/netExport-0.8b9.xpi")); p.addExtension(new File("c:/firestarter-0.1.a5.xpi"));} catch (IOException e) { throw new RuntimeException(“Failed to load extensions:", e);}p.setPreference("extensions.firebug.netexport.autoExportActive", true);//p.setPreference("extensions.firebug.netexport.defaultLogDir", "c:/");p.setPreference("extensions.firebug.onByDefault", true);p.setPreference("extensions.firebug.defaultPanelName", "net");p.setPreference("extensions.firebug.net.enableSites", true);p.setPreference("extensions.firebug.previousPlacement", 1);driver = new FirefoxDriver(p);

Method 3: Example - Firebug + NetExport

Method 4: Example - Firebug + NetExport

Method 4: Use a Proxy

Use built in Selenium 1 proxy

Use a 3rd party proxy library Many available, few capture metrics in a

convenient way Two good ones:

BrowserMob Proxy Fiddler

Method 4: Selenium built-in Proxy

Use API Selenium.captureNetworkTraffic()

Selenium selenium = new DefaultSelenium("localhost", 4444, "*firefox", "http://www.webmetrics.com"); selenium.start("captureNetworkTraffic=true");selenium.open("http://www.webmetrics.com"); String json = selenium.captureNetworkTraffic("json"); selenium.stop();

Method 4: Selenium built-in Proxy Not in HTTP Archive (HAR) format, but does

report: HTTP status code, URL and HTTP method Request & response headers Overall request->response timing Bytes downloaded

Works with Chrome and Firefox Doesn’t work for WebDriver tests.

Still work with Selenium 2 RC, but only if send jobs via Selenese API.

Not much control over the proxy

Method 4: Advantages of using a Proxy Testing benefits beyond performance

monitoring Blacklisting/whitelisting URLs

Comparing page load times of site with/without external content

Not hitting analytics/advertising content Simulating external content being offline

URL rewrites Redirect production URLs back to staging/test

environment Pretend to be production as far as the browser is

concerned (e.g. for cookies) Make sure ad/metrics requests are being made

Method 4: Advantages of using a Proxy Header changes

Set the user agent manually to test different browser behavior

Auto authentication

Wait until all content is downloaded HTTP idle for X seconds

Limit bandwidth

Method 4: BrowserMob Proxy

Open source cross platform proxy written in Java. HTTP Archive support Friendly API

Source code available: https://github.com/lightbody/browsermob-prox

y

Method 4: BrowserMob Proxy

Export HAR Sample

import org.browsermob.proxy.ProxyServer;

ProxyServer proxy = new ProxyServer(9090);proxy.start();

Proxy mobProxy = new Proxy().setHttpProxy("localhost:9090");

DesiredCapabilities caps = new DesiredCapabilities();caps.setCapability("proxy", mobProxy);

FirefoxDriver driver = new FirefoxDriver(caps);

Method 4: BrowserMob Proxy

Run test, denoting pages in HAR

proxy.newHar("Yahoo");driver.get("http://yahoo.com");proxy.endPage();

proxy.newPage("CNN");driver.get("http://cnn.com");proxy.endPage();

Method 4: BrowserMob Proxy

Write out HTTP Archive file

proxy.getHar().writeTo(new File("test.har"));

Method 4: BrowserMob Proxy

Redirecting URLs

proxy.blacklistRequests(regex, responseCode)proxy.whitelistRequests(regex, responseCode)

Blacklist/Whitelist URLs

proxy.rewriteUrl(regex, replace)

Limit Bandwidth

proxy.setDownstreamKbps(kbps)proxy.setUpstreamKbps(kbps)

Method 4: BrowserMob Proxy

When to use Cross platform Java Can set the browser proxy

Method 4: Proxy - FiddlerCore Fiddler is an application for viewing HTTP

traffic on Windows. Works with Chrome, FireFox, Internet Explorer. Fiddler Application is built on FiddlerCore

.NET library Allows extensive programmatic control over

the proxy Captures HTTP timings Allows request/responses to be intercepted and modified

Configures itself as default Windows proxy (supports proxy chaining)

Can decrypt SSL traffic

To start the proxy and register it as the default system wide proxy.

Each HTTP transaction can then be recorded as follows:

Method 4: Proxy - FiddlerCore

// Initialize the proxyFiddler.FiddlerApplication.Startup(

8877, FiddlerCoreStartupFlags.Default);

var items = new List<Fiddler.Session>(); Fiddler.FiddlerApplication.AfterSessionComplete += delegate(Fiddler.Session oS){

items.Add(oS);};

Method 4: Proxy - FiddlerCore Run Selenium test as normal

As each item is downloaded it will be added to the ’items’ var in previous slide.

string baseUrl = "http://www.webmetrics.com";var webDriver =

new OpenQA.Selenium.Firefox.FirefoxDriver();var selenium =

new Selenium.WebDriverBackedSelenium(webDriver, baseUrl);

selenium.Start();selenium.Open(baseUrl);selenium.WaitForPageToLoad("30000");selenium.Stop();

Method 4: Proxy (Fiddler -> HAR) Using the HAR Exporter to convert the

sessions to HAR Add FiddlerCore-BasicFormats.dll to the project

references and load the assembly: https://www.fiddler2.com/dl/FiddlerCore-BasicFormats

.zip

String exePath = Assembly.GetExecutingAssembly().Location;String path = Path.Combine( Path.GetDirectoryName(exePath), @"FiddlerCore-BasicFormats.dll");

FiddlerApplication.oTranscoders.ImportTranscoders(path);

Method 4: Proxy (Fiddler -> HAR) Finally, export the HAR to a file:

var oExportOptions = new Dictionary<string, object>();string filename = @"output.har”;oExportOptions.Add("Filename", filename);

Fiddler.FiddlerApplication.DoExport( "HTTPArchive v1.2", sessions, oExportOptions, null);

Method 4: Proxy Bonus Features Modifying HTTP requests

Fiddler.FiddlerApplication.BeforeRequest +=delegate(Fiddler.Session oS)

{// Ignore requests made to the main site.Regex matchUrl = new Regex("webmetrics\\.com”);if (matchUrl.IsMatch(oS.fullUrl)) {

oS.utilCreateResponseAndBypassServer();oS.responseCode = 200;

}};

Fiddler.FiddlerApplication.BeforeResponse

In the same way responses from the server can be modified

Method 4: Proxy – FiddlerCore

When to use Windows Only Cross browser .NET

Method 5: Wire HTTP capture Captured network traffic can be converted to HAR

On Unix/OSX use tcpdumptcpdump -i en0 -n -s 0 -w traffic.pcap

On Windows use WinPCapwinpcap -i 0 -n -s 0 -w traffic.pcap

Then use pcap2har to do the conversion:main.py traffic.pcap http-requests.har

Pcap2har https://github.com/andrewf/pcap2har

Method 5: Wire HTTP capture Captures all HTTP traffic Timings based on actual network traffic No interference from proxy

No extra network delays/context switches talking to the proxy

Browsers sometimes behave differently when talking to a proxy

Summary Discussed 5 methods for recording

performance metrics Use timers around selenium commands Use WebTimings JavaScript API Firebug & NetExport Proxy

Use the built-in Selenium proxy Use a 3rd party proxy

Sniff network traffic

Links BrowserMob proxy

https://github.com/lightbody/browsermob-proxy

Fiddler Cookbook http://www.fiddler2.com/fiddler/dev/ScriptSamples.

asp

Examples from this talk https://github.com/watsonmw/selenium-pageload

metrics