Paruj Ratanaworabhan,
Ben Livshits, and Ben Zorn
JSMeter: Characterizing the Behavior of JavaScript Web Applications
1
In collaboration with
David Simmons, Corneliu Barsan, and Allen Wirfs-Brock
Why Measure JavaScript?
• Standardized, de facto language for the web
– Support in every browser, much existing code
• Major Microsoft investment
– Web Application Companion based on Script#
• Benchmarks are driving browser marketing: “browser wars”
– Are benchmarks representative?
– Limited understanding of its behavior in real sites
• Who cares?
– Users, web site designers, JavaScript engine developers
2
3
ZDNet 29 May 2008
ghacks.net Dec. 2008
Artificial Benchmarks versus Real World Sites
4
Maps Maps
7 V8 programs:
• richards
• deltablue
• crypto
• raytrace
• earley-boyer
• regexp
• splay
8 SunSpider programs:
• 3-draytrace
• access-nbody
• bitops-nsieve
• controlflow
• crypto-aes
• date-xparb
• math-cordic
• string-tagcloud
JSMeter
Goals of this Talk
• Show behavior of JavaScript benchmarks versus real sites
• Consider how benchmarks can mislead design decisions
11 real sites
5
Instrumentation Framework
\ie\jscript\*.cpp
Source-level instrumentation
custom jscript.dll
custom trace fileswebsite visits
Offline analyzers
custom trace files0%
10%20%30%40%50%60%70%80%90%
100%
Constant
Other Str Ops
Concat Op
Visiting the Real Sites
• Attempted to use each site in “normal” way:
6
amazon Search a book, add to shopping cart, sign in, and sign outbing Type in a search query and also look for images and newsbingmap Search for a direction from one city to anothercnn Read front page newsebay Search for a notebook, bid, sing in, and sign outeconomist Read front page news, view comments
facebookLog in, visit a friend pages, browse through photos and comments
gmail Sign in, check inbox, delete a mail, and sign outgoogle Type in a search query and also look for images and newsgooglemap Search for a direction from one city to anotherhotmail Sign in, check inbox, delete a mail, and sign out
Understanding JavaScript Behavior
Code
Events
JavaScript
Objects
7
Code Behavior
Code
Events
JavaScript
Objects
8
• Function size
• Instructions/call
• Code locality
• Instruction mix
Total Bytes of JavaScript Source
9
amaz
on
bingm
apeb
ay
face
book
hotm
ail
richar
ds
cryp
to
earle
y
splay
3d-rayt
race
bitops-
nsieve
cryp
to-a
es
mat
h-cor
dic
strin
g-tagclo
ud
0
500
1000
1500
2000
2500
So
urc
e s
ize
(k
ilo
byte
s)
Real Sites V8 SunSpider
Code|Objects|Events
Static Unique Functions Executed
11
amaz
on
bing
map
ebay
face
book
goog
le
hotm
ail
richa
rds
cryp
to
earle
ysp
lay
3d-ra
ytra
ce
bito
ps-n
sieve
cryp
to-a
es
mat
h-co
rdic
strin
g-ta
gcloud
0
500
1,000
1,500
2,000
2,500
3,000
3,500
4,000
Uniq
ue F
unct
ions E
xecu
ted
Real Sites V8 SunSpider
Code|Objects|Events
Total Bytecodes Executed
12
amaz
on
bingm
apeb
ay
face
book
hotm
ail
richar
ds
cryp
to
earle
y
splay
3d-rayt
race
bitops-
nsieve
cryp
to-a
es
mat
h-cor
dic
strin
g-tagclo
ud
0
5000
10000
15000
20000
25000
30000Tota
l B
yte
codes E
xcu
ted (
thousands)
90395
Real Sites V8 SunSpider
Code|Objects|Events
Bytecodes / Call
14am
azon
bingm
apeb
ay
face
book
hotm
ail
richar
ds
cryp
to
earle
y
splay
3d-rayt
race
bitops-
nsieve
cryp
to-a
es
mat
h-cor
dic
strin
g-tagclo
ud
0.00
50.00
100.00
150.00
200.00
250.00
300.00
Byte
co
de
s /
Ca
ll
function(a,b) { var i=0,elem,pos=a.length; if(D.browser.msie) { while(elem=b[i++]) if(elem.nodeType!=8) a[pos++]=elem; } else while(elem=b[i++]) a[pos++]=elem; return a}
Real Sites V8 SunSpider
Code|Objects|Events
Fraction of Code Executed
15
amaz
on
bingm
apeb
ay
face
book
hotm
ail
richar
ds
cryp
to
earle
y
splay
3d-rayt
race
bitops-
nsieve
cryp
to-a
es
mat
h-cor
dic
strin
g-tagclo
ud
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
90.00%
100.00%
Fra
cti
on
of
Co
de
Exe
cu
ted
Most codenot executed
Real Sites V8 SunSpider
Code|Objects|Events
Hot Function Distribution
16
Real Sites V8 Benchmarks
80% of time in < 10 functions80% of time in 100+ functions
80% 80%
Code|Objects|Events
Opcode Distribution
17Green = SunSpider | Blue= Real Web Apps | Red = V8
Real Apps
Outliers
Code|Objects|Events
Object Allocation Behavior
Code
Events
JavaScript
Objects
18
• Allocation by types
• Lifetime distribution
• Live heap composition
Total Bytes Allocated
19
amaz
on
bingm
apeb
ay
face
book
hotm
ail
richar
ds
cryp
to
earle
y
splay
3d-rayt
race
bitops-
nsieve
cryp
to-a
es
mat
h-cor
dic
strin
g-tagclo
ud
0
5000
10000
15000
20000
25000
30000
35000
40000
45000Tota
l h
eap
data
(kilob
yte
s)
Real Sites V8 SunSpider
Code|Objects|Events
Heap Data by Type
20
amazon
bingmapebay
hotmail
richard
s
crypto
earley
splay
access-
nbody
contro
lflow
date-xparb
regexp-dna
aggregate0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
ObjectsEnumaratorRegExpMathErrorDateNative FuncStringArraysScript FuncH
eap
Dat
a by
Typ
e
Real Sites V8 SunSpider
Many functionsRest are strings
Few benchmarksallocate much data
Code|Objects|Events
Object Type Distribution
21
Real Apps
Code|Objects|Events
economistis an outlier(arrays)
Live Heap Over Time (gmail)
22
Functions growsteadily
Code|Objects|Events
GC reducessize of heap
Objects growsteadily too
Live Heap over Time (ebay)
23
Heaps repeatedly created, discarded
Heap containsmostly functions
Code|Objects|Events
Heap drops to 0 on page load
Two Search Websites
24
Bing(Web 2.0)
Google(Web 1.0)
Code|Objects|Events
Two Mapping Websites
25
Bing Maps Google Maps
Code|Objects|Events
Object Lifetimes (V8)
26
earley splay
Code|Objects|Events
Lives forever
Dies immediately
Lives untilprogram ends
Object Lifetimes (gmail)
27
Strings areshortest lived
Functions arelongest lived
Many objectslive a major fraction ofpage lifetime
This is as oldas an objectcan be
Code|Objects|Events
Event Handlers in JavaScript
Code
Events
JavaScript
Objects
28
• Number of events• Sizes of handlers
Event-driven Programming Model
• Single-threaded, non-preemptive event handlers
• Example of some standard event-handler attributes:
– onabort, onclick, and onmouseover
• Very different from batch processing of benchmarks
• Handler responsiveness critical to user experience
29
Code|Objects|Events
Total Events Handled
30
amaz
onbin
g
bingm
ap cnn
ebay
econ
omist
face
book
gmail
map
hotm
ail
richar
ds
delta
blue
cryp
to
rayt
race
earle
y
regex
psp
lay
0
1,000
2,000
3,000
4,000
5,000
6,000
7,000T
ota
l E
ve
nts
Ha
nd
led
Real Sites V8
Almost no events
Code|Objects|Events
Median Bytecodes / Event Handled
32am
azon
bing
bing
map cn
neb
ay
econ
omist
face
book
gmail
goog
le
goog
lem
ap
hotm
ail
0
50
100
150
200
250
300
350
400
450
500M
ed
ian
Byte
co
de
s /
Eve
nt
Ha
nd
led
Code|Objects|Events
Maximum Bytecodes / Event Handled
33am
azon
bing
bing
map cn
neb
ay
econ
omist
face
book
gmail
goog
le
goog
lem
ap
hotm
ail
0
200,000
400,000
600,000
800,000
1,000,000
1,200,000M
axim
um
Byte
co
de
s /
Eve
nt
Ha
nd
led
Code|Objects|Events
Distribution of Time in Handlers
35
Code|Objects|Events
A Simple Experiment• Observation
– Real web apps have lots of code (much of it cold)– Benchmarks do not
• Question: What happens if the benchmarks have more code?– We added extra, unused to code to 6 SunSpider
benchmarks– We measured the impact on the benchmark
performance
36
Performance Impact of Cold Code
37
3d-raytr
ace
access
-nbody
bitops-n
sieve
contro
lflow
crypto-ae
s
math-co
rdic
string-t
agclo
ud0
100
200
300
400
500
600
700
800
900
0K200K400K800K1M2M
Tim
e (m
sec)
3d-raytr
ace
access
-nbody
bitops-n
sieve
contro
lflow
crypto-ae
s
math-co
rdic
string-t
agclo
ud0
100
200
300
400
500
600
700
800
900
0K200K400K800K1M2M
Tim
e (m
sec)
Chrome IE 8
Cold code has non-uniform impacton execution time
Without cold code, Chrome12x faster than IE8With 2M cold code,Chrome 4.7x faster than IE8Which comparison is more meaningful?
Cold code makesSunSpider on Chromeup to 4.5x slower
Impact of Benchmarks
• What gets emphasis– Making tight loops fast– Optimizing small amounts of code
• Important issues ignored– Garbage collection (especially of strings)– Managing large amounts of code– Optimizing event handling – Considering JavaScript context between page loads
38
Conclusions
• JSmeter is an instrumentation framework
– Used to measure and compare JavaScript applications
– High-level views of behavior promote understanding
• Behavior in benchmarks significantly differs from real sites
– Misleads designers, skews implementations
• Next steps
– Develop and promote better benchmarks
– Design and evaluate better JavaScript runtimes
– Promote better performance tools for JavaScript developers
39
Additional Resources• Project
– http://research.microsoft.com/en-us/projects/jsmeter/ • Papers
– "JSMeter: Measuring JavaScript Behavior in the Wild", Paruj Ratanaworabhan, Benjamin Livshits, David Simmons, and Benjamin Zorn, MSR-TR-2010-8, January 2010 (17 pages)
– "JSMeter: Characterizing Real-World Behavior of JavaScript Programs", Paruj Ratanaworabhan, Benjamin Livshits, David Simmons, and Benjamin Zorn, MSR-TR-2009-173, December 2009 (49 pages)
– “An Analysis of the Dynamic Behavior of JavaScript Programs”, Gregor Richards, Sylvain Lebresne, Brian Burg, Jan Vitek (to appear in PLDI 2010)
40