1 Rep
rod
uct
ion inte
rdit
e. ©
ip
-lab
el 2
01
2
DO ALL USERS BENEFIT EQUALLY FROM
WEB PERFORMANCE OPTIMIZATIONS?
Arnaud Becart – @arnaud_beip-label
2 Rep
rod
uct
ion inte
rdit
e. ©
ip
-lab
el 2
01
2REAL-USER MONITORING ≠
SYNTHETIC TESTING
Previously, at Velocity Berlin…
4 Rep
rod
uct
ion inte
rdit
e. ©
ip
-lab
el 2
01
2
PROOF BY EXAMPLE
Mobile + 0 sTablet + 4 s
PC + 4 s Synthetic
Tests+ 0 s
Real Users
Synthetic Tests don’t see everything
5 Rep
rod
uct
ion inte
rdit
e. ©
ip
-lab
el 2
01
2
SOME DIFFERENCES BETWEEN SYNTHETIC TESTS & RUM
Browser
Browser version
OS
Device
CPUMemory
OS version
Fixed location
Fixed last mile
Bandwidth
Provider Browser add-ons
Antivirus
Mobilelast mile
2. No synthetic testing
Networkaccess
3. Variable, depending on users’ behaviour
Nomadic usage
1. Similar or different between synthetic & real-user testing
Browser cache
Multiple tabs
6 Rep
rod
uct
ion inte
rdit
e. ©
ip
-lab
el 2
01
2
BACK TO THE TITLE
DO ALL USERS BENEFIT EQUALLY FROMWEB PERFORMANCE
OPTIMIZATIONS?
Fewer HTTP
requests
Expires header
CSS at the top Accelerate
rendering
CDNGzip
JS at the bottom
Reduce size
…(thanks Steve)
Defer third-party
content
Avoid redirects
8 Rep
rod
uct
ion inte
rdit
e. ©
ip
-lab
el 2
01
2
METHODOLOGY
synthetic_monitoring()optimization ()
while not improved(synthetic_monitoring()):optimization()
//responsetime is better!real_user_monitoring()
9 Rep
rod
uct
ion inte
rdit
e. ©
ip
-lab
el 2
01
2
SYNTHETIC TREND
Fewer requests
& Compression
35%Reduction
in page size
30% quicker start rendering
23% quicker page load time
Page Size
Page Load Time
2.5 Mb
1.6 Mb
7.4 s
5.7 s
1st Step: Confirm trend with synthetic testing
10 Rep
rod
uct
ion inte
rdit
e. ©
ip
-lab
el 2
01
2
Real Users’ Performance gain: 11%
WHAT ABOUT REAL USERS ?
2nd Step: Align Real Users & Synthetic tests
URL, Country, ISP, Metric (onload event), Period (2 months)
Page load time: 9.2 s 8.2 s
3rd Step: Analyze 20M real-user measurements
11 Rep
rod
uct
ion inte
rdit
e. ©
ip
-lab
el 2
01
2
BACK TO THE TITLE AGAIN
DO ALL USERS BENEFIT EQUALLY FROM WEB PERFORMANCE
OPTIMIZATIONS?
12 Rep
rod
uct
ion inte
rdit
e. ©
ip
-lab
el 2
01
2
USER SATISFACTION
% of users loading the page in less than 9 seconds
Mobile: 43% 55%Tablet: 52% 60%
+12
+8
13 Rep
rod
uct
ion inte
rdit
e. ©
ip
-lab
el 2
01
2
THE DEVICE EFFECT
Smartphones = High BenefitTablets = Small Benefit
Mobile
PC
Probe
Tablet
0.0% 5.0% 10.0% 15.0% 20.0% 25.0% 30.0%
+24.6%
+10.6%
+22.7%
+6.7%
Response time gain (%) per device
12.1 11.3 s
7.4 5.7 s
9.1 8.1 s
21.7 16.4 s
14 Rep
rod
uct
ion inte
rdit
e. ©
ip
-lab
el 2
01
2
THE NETWORK EFFECT
Mobile
Tablet
PC
0.0% 5.0% 10.0%15.0%20.0%25.0%30.0%
20.1%
6.9%
10.4%
29.7%
17.6%
15.0%
Mobile NetworkFixed Network
Response time gain (%) per type of Network connection
Slow bandwidth + fewer bytes to load = High Benefit
15 Rep
rod
uct
ion inte
rdit
e. ©
ip
-lab
el 2
01
2
THE ‘CPU’ EFFECT
*Not just CPU: processing capacity & available memoryDevices with Very Good or Very Bad Performance
(old?)benefit less from size optimization
Very Old
Average
Very Good
+12.1%
+18.0%
+15.2%
+7.9%
+4.2%
+2.9%Dual Core 2.2 GHz 4G RAM
iPad 1
iPhone 4
5.5 5.3 s
7.5 7 s
8.5 8 s
10 8.5 s
14.5 12
s
20s 17 s
iPhone 5
Response time gain (%) per device capacity*
16 Rep
rod
uct
ion inte
rdit
e. ©
ip
-lab
el 2
01
2
THE BROWSER EFFECT
No improvement for SafariHigher benefit for IE
-5.0% 0.0% 5.0% 10.0% 15.0% 20.0%
+16.0%
+10.6%
+6.4%
-0.9%8.5 8.6 s
9.9 9.3 s
8.3 7.5 s
9 7.6 s
Response time gain (%) per browser on fixed devices
17 Rep
rod
uct
ion inte
rdit
e. ©
ip
-lab
el 2
01
2
THE BROWSER EFFECT
Very Old
Old
Average
Good
Very Good
Premium Devices!
-10.0% -5.0% 0.0% 5.0% 10.0% 15.0% 20.0%
SafariChromeFirefoxIE
Chrome: 22.8 23.3 s
Safari: 7.8 8.4 sSafari: 8.3 8.4 s
Response time gain (%) per browser & processing (PC)
Are Firefox users geeks?Is Safari not made for new devices?
Is Chrome not made for very old devices?
Firefox: 6 5.3 s
18 Rep
rod
uct
ion inte
rdit
e. ©
ip
-lab
el 2
01
2
CONCLUSIONDO ALL USERS BENEFIT EQUALLY FROM
WEB PERFORMANCE OPTIMIZATIONS?
19 Rep
rod
uct
ion inte
rdit
e. ©
ip
-lab
el 2
01
2
CONCLUSIONDO ALL USERS BENEFIT EQUALLY FROM
WEB PERFORMANCE OPTIMIZATIONS ?Synthetic
Tests+23%
Real Users+11%
PC+10.6
%
IE +18%
Smartphone
+25%
Old PC+18%
Tablet+7%
New PC+4%
Smartphone 3G Network
+30%Tablet
3G Network+18%
Chrome +8%
20 Rep
rod
uct
ion inte
rdit
e. ©
ip
-lab
el 2
01
2
Don’t trust only Synthetic Tests to justify web optimizations
Not all devices are equal – especially with the network factorNot all browsers are equal – especially with “CPU”/”Old device” factor
Don’t be too quick to draw conclusions about the actual effectiveness of optimizations for your real users