Upload
philip-tellis
View
1.258
Download
1
Tags:
Embed Size (px)
DESCRIPTION
Why, How and What to measure in terms of end user perceived performance
Citation preview
Why measure?boomerang
data data data
Measuring the web with boomerang
Philip Tellis / [email protected]
SF Performance Meetup / 2010-08-11
SF Performance Meetup / 2010-08-11 Measuring the web with boomerang
Why measure?boomerang
data data data
$ finger philip
Philip [email protected]
@bluesmoonyahoogeekhttp://bluesmoon.info/
SF Performance Meetup / 2010-08-11 Measuring the web with boomerang
Why measure?boomerang
data data data
The slow webMeasurementsMeasuring with javascript
Where does all the time go?
SF Performance Meetup / 2010-08-11 Measuring the web with boomerang
Why measure?boomerang
data data data
The slow webMeasurementsMeasuring with javascript
Who controls it?
Some of this we control and some of it we don’t
SF Performance Meetup / 2010-08-11 Measuring the web with boomerang
Why measure?boomerang
data data data
The slow webMeasurementsMeasuring with javascript
Back end
Measuring and improving back end performance can be doneduring development
SF Performance Meetup / 2010-08-11 Measuring the web with boomerang
Why measure?boomerang
data data data
The slow webMeasurementsMeasuring with javascript
80-20
Turns out that less than 20% of the time is spent on the backend
SF Performance Meetup / 2010-08-11 Measuring the web with boomerang
Why measure?boomerang
data data data
The slow webMeasurementsMeasuring with javascript
Front end
It’s what we can’t control that bites us
SF Performance Meetup / 2010-08-11 Measuring the web with boomerang
Why measure?boomerang
data data data
The slow webMeasurementsMeasuring with javascript
Too many variations
browsers
SF Performance Meetup / 2010-08-11 Measuring the web with boomerang
Why measure?boomerang
data data data
The slow webMeasurementsMeasuring with javascript
Too many variations
plugins
SF Performance Meetup / 2010-08-11 Measuring the web with boomerang
Why measure?boomerang
data data data
The slow webMeasurementsMeasuring with javascript
Too many variations
OSes
SF Performance Meetup / 2010-08-11 Measuring the web with boomerang
Why measure?boomerang
data data data
The slow webMeasurementsMeasuring with javascript
Too many variations
viruses
SF Performance Meetup / 2010-08-11 Measuring the web with boomerang
Why measure?boomerang
data data data
The slow webMeasurementsMeasuring with javascript
Too many variations
antiviruses
SF Performance Meetup / 2010-08-11 Measuring the web with boomerang
Why measure?boomerang
data data data
The slow webMeasurementsMeasuring with javascript
Too many variations
microwaves
SF Performance Meetup / 2010-08-11 Measuring the web with boomerang
Why measure?boomerang
data data data
The slow webMeasurementsMeasuring with javascript
Too many variations
baby monitors
SF Performance Meetup / 2010-08-11 Measuring the web with boomerang
Why measure?boomerang
data data data
The slow webMeasurementsMeasuring with javascript
Too many variations
naughty neighbours
SF Performance Meetup / 2010-08-11 Measuring the web with boomerang
Why measure?boomerang
data data data
The slow webMeasurementsMeasuring with javascript
Too many variations
file shares
SF Performance Meetup / 2010-08-11 Measuring the web with boomerang
Why measure?boomerang
data data data
The slow webMeasurementsMeasuring with javascript
Too many variations
governments
SF Performance Meetup / 2010-08-11 Measuring the web with boomerang
Why measure?boomerang
data data data
The slow webMeasurementsMeasuring with javascript
Too many variations
rodents
SF Performance Meetup / 2010-08-11 Measuring the web with boomerang
Why measure?boomerang
data data data
The slow webMeasurementsMeasuring with javascript
Too many variations
Try simulating all that in the lab!
SF Performance Meetup / 2010-08-11 Measuring the web with boomerang
Why measure?boomerang
data data data
The slow webMeasurementsMeasuring with javascript
We need to measure real end-user performance
SF Performance Meetup / 2010-08-11 Measuring the web with boomerang
Why measure?boomerang
data data data
The slow webMeasurementsMeasuring with javascript
We need to measure it from the real end-user’s box
SF Performance Meetup / 2010-08-11 Measuring the web with boomerang
Why measure?boomerang
data data data
The slow webMeasurementsMeasuring with javascript
Ask the user?
SF Performance Meetup / 2010-08-11 Measuring the web with boomerang
Why measure?boomerang
data data data
The slow webMeasurementsMeasuring with javascript
Bias
While this might work, it isn’t necessarily representative
SF Performance Meetup / 2010-08-11 Measuring the web with boomerang
Why measure?boomerang
data data data
The slow webMeasurementsMeasuring with javascript
A/B testing
You also want to be able to dynamically tune which users getwhich tests
SF Performance Meetup / 2010-08-11 Measuring the web with boomerang
Why measure?boomerang
data data data
The slow webMeasurementsMeasuring with javascript
Phone home
It’s most useful if you can send these measurements back toyour server for analysis
SF Performance Meetup / 2010-08-11 Measuring the web with boomerang
Why measure?boomerang
data data data
The slow webMeasurementsMeasuring with javascript
Mostly ubiquitous
We know that javascript is available on almost every browser
SF Performance Meetup / 2010-08-11 Measuring the web with boomerang
Why measure?boomerang
data data data
The slow webMeasurementsMeasuring with javascript
Rich pages
We really want to measure the performance of rich pageswhich depend on javascript already
SF Performance Meetup / 2010-08-11 Measuring the web with boomerang
Why measure?boomerang
data data data
The slow webMeasurementsMeasuring with javascript
Limited
But javascript can’t measure everything... we get as close aswe can
SF Performance Meetup / 2010-08-11 Measuring the web with boomerang
Why measure?boomerang
data data data
What?How does it work?Accuracy
A piece of javascript that you add to your web page where itmeasures and beacons back to you the end user’s perceivedperformance of your page
SF Performance Meetup / 2010-08-11 Measuring the web with boomerang
Why measure?boomerang
data data data
What?How does it work?Accuracy
How?
<script src="boomerang.js" type="text/javascript"></script><script type="text/javascript">BOOMR.init({user_ip: "<user’s ip address>",beacon_url: "http://yoursite.com/path/to/beacon.php"});</script>
SF Performance Meetup / 2010-08-11 Measuring the web with boomerang
Why measure?boomerang
data data data
What?How does it work?Accuracy
What does it do?
About once a week, measures user’s bandwidth andlatency to your serverOn (almost) every request, measures the time it took toload the current pageBeacons these results back to your serverOther stuff based on plugins
SF Performance Meetup / 2010-08-11 Measuring the web with boomerang
Why measure?boomerang
data data data
What?How does it work?Accuracy
How does it do it?
Let’s take that one at a time
SF Performance Meetup / 2010-08-11 Measuring the web with boomerang
Why measure?boomerang
data data data
What?How does it work?Accuracy
How do we measure latency?
Download a 32 byte gif 10 times in sequenceMeasure the time to download eachDiscard the first measurement because it’s overpricedCalculate the arithmetic mean, standard deviation andmargin of error of the remaining
SF Performance Meetup / 2010-08-11 Measuring the web with boomerang
Why measure?boomerang
data data data
What?How does it work?Accuracy
How do we measure bandwidth?
After the latency test is done, we download progressivelylarger imagesStop at the first image that times outRedownload that image a few more timesCalculate the median, standard deviation and margin oferror of the largest images
SF Performance Meetup / 2010-08-11 Measuring the web with boomerang
Why measure?boomerang
data data data
What?How does it work?Accuracy
How do we measure page load time?
In the onbeforeunload event, measure the time andstore it in a cookieIn the onload event, check the cookie, and measure thedifference with the current timeWe also make sure that the page that set the cookie is thereferrer of the current page
SF Performance Meetup / 2010-08-11 Measuring the web with boomerang
Why measure?boomerang
data data data
What?How does it work?Accuracy
What? Two pages?
Yes, this needs two pages and cookies. If those aren’tsupported, we try to use the WebTiming API.
SF Performance Meetup / 2010-08-11 Measuring the web with boomerang
Why measure?boomerang
data data data
What?How does it work?Accuracy
How accurate is it?
Latency measurements are very accurate (±1%)Bandwidth is to an order of magnitude. For badconnections can be ±30%
Page load time sometimes has outliers, you needpost-filteringThe margin of error tells you how good your data is
SF Performance Meetup / 2010-08-11 Measuring the web with boomerang
Why measure?boomerang
data data data
FilteringGroupingMore...
What do we do with the data?
Sanity checking to:Remove fake dataRemove abusive dataMaybe just rate limiting
SF Performance Meetup / 2010-08-11 Measuring the web with boomerang
Why measure?boomerang
data data data
FilteringGroupingMore...
What do we do with the data?
Statistical analysis to:Remove outliersAggregate based on bandwidth blocksMeasure trends over time and correlate them with codechanges
SF Performance Meetup / 2010-08-11 Measuring the web with boomerang
Why measure?boomerang
data data data
FilteringGroupingMore...
Bandwidth slabs
0-100 kbps100-300 kbps300-2000 kbps2-6 Mbps6+ Mbps
SF Performance Meetup / 2010-08-11 Measuring the web with boomerang
Why measure?boomerang
data data data
FilteringGroupingMore...
Bandwidth slabs
Group page load times based on bandwidth slab
SF Performance Meetup / 2010-08-11 Measuring the web with boomerang
Why measure?boomerang
data data data
FilteringGroupingMore...
Bandwidth slabs
Data points from some countries may require smaller slabs
SF Performance Meetup / 2010-08-11 Measuring the web with boomerang
Why measure?boomerang
data data data
FilteringGroupingMore...
Geographic data
Looking at latency from different geographic locations can tellyou where to put your next CDN
SF Performance Meetup / 2010-08-11 Measuring the web with boomerang
Why measure?boomerang
data data data
FilteringGroupingMore...
ISPs
Grouping data by ISP can tell you who’s behaving badly
SF Performance Meetup / 2010-08-11 Measuring the web with boomerang
Why measure?boomerang
data data data
FilteringGroupingMore...
More data
Write plugins to get more performance dataWe already have a DNS pluginI’m thinking of an IPv6 v/s IPv4 pluginWhat about a full WebTiming plugin?
SF Performance Meetup / 2010-08-11 Measuring the web with boomerang
Why measure?boomerang
data data data
FilteringGroupingMore...
You decide
Once you have the data, you can do anything with it
SF Performance Meetup / 2010-08-11 Measuring the web with boomerang
Why measure?boomerang
data data data
FilteringGroupingMore...
Thank you
SF Performance Meetup / 2010-08-11 Measuring the web with boomerang
Why measure?boomerang
data data data
FilteringGroupingMore...
Photo credits
flickr.com/photos/21233184@N02/4389412851
SF Performance Meetup / 2010-08-11 Measuring the web with boomerang
Why measure?boomerang
data data data
FilteringGroupingMore...
Contact me
Philip Tellisyahoogeek@bluesmoonhttp://bluesmoon.info/slideshare.net/[email protected]
SF Performance Meetup / 2010-08-11 Measuring the web with boomerang
Why measure?boomerang
data data data
FilteringGroupingMore...
References
github.com/yahoo/boomerangMore bandwidth doesn’t matter (much) – Mike BelsheAnalysing Bandwidth & Latency – YUI BlogIt’s the latency, stupid – Stuart CheshireThe statistics of web performance
SF Performance Meetup / 2010-08-11 Measuring the web with boomerang