Upload
mahala
View
47
Download
0
Tags:
Embed Size (px)
DESCRIPTION
Measuring Interactive Performance with VNCplay. Nickolai Zeldovich, Ramesh Chandra Stanford University. Measuring performance. Scientific computing or server workloads: How long does it take for my program to run? Measure total running time. Measuring performance. - PowerPoint PPT Presentation
Citation preview
Measuring InteractivePerformance with VNCplay
Nickolai Zeldovich, Ramesh Chandra
Stanford University
Measuring performance
• Scientific computing or server workloads:– How long does it take for my program to run?
• Measure total running time
Measuring performance
• Users of interactive applications:– I asked for something, when will I get it?
• But we still measure total runtime– WinBench runs Word, PowerPoint, etc as fast
as possible, and measures total runtime
We are measuring the wrong thing
Case in point
• Use Microsoft PowerPoint for 5 minutes
• Two machines: 2.0 GHz and 300 MHz– Expect to see different performance!
• Two performance metrics:– Total runtime: from start to finish– Response time: from click to output on screen
Performance Metrics
0
100
200
300
400
To
tal
Ru
nti
me
(sec
on
ds)
300 MHz 2.0 GHz0
500
1000
1500
2000
Med
ian
Res
po
nse
Tim
e (m
sec)
300 MHz 2.0 GHz
Total Runtime Response Time
Users pause a lot
. . .
0 sec 0.1 sec 0.2 sec 5 sec. . .
Response time is noise in total runtime
Outline
• Overview of VNCplay
• Current approaches
• How VNCplay works
• Evaluation– Quantitative results– Qualitative experience
• Summary
Overview of VNCplay
• To compare interactive performance, we need reproducible workloads– Record interactive user session– Replay session in different environments
• Compare response time between replays
Current approaches
Display protocols: X11, VNC, Windows
Key strokes, mouse clicks
Screen updates
Current approaches
Screen updates
. . .0 sec 1 sec 5 sec 8 secTime
InputEvent
Recorder
Tools like Xnee, Visual Test, AutoIt, rfbplaymacro
Current approaches
Screen updates
. . .0 sec 1 sec 5 sec 8 secTime
InputEvent
Replayer
What if system is slow?
0 sec 0.5 sec 1.0 sec
Need feedback of when the system responds
1.5 sec Time
What should happen?
0 sec 0.5 sec 1.0 sec
Need feedback of when the system responds
1.5 sec Time
Workarounds?
• Current tools allow user to insert delay statements to compensate for slowness
– Creates unrealistic user behavior
– Tedious and error-prone
VNCplay approach
• Big problem is mouse clicks
• Take a screen snapshot at each mouse click during recording
• Wait for the same thing during replay
VNCplay approach
0 sec 1 sec 5 sec 8 secTime
InputEvent
Screen updates
Recorder
VNCplay approach
Screen updates
0 sec 1 sec 5 sec 8 secTime
InputEvent
Replayer
VNCplay details
• Only snapshot the screen around the cursor– Reduces trace file size– Ignores irrelevant things (e.g. system clock)
• Other tricks described in the paper
Analyzing Replays
Time
Response Time
System 1:
System 2:
Match
Evaluation
• OpenOffice on Linux
• Scaled CPU frequency from 2.0 GHz down to 300 MHz
• Compare total running time and interactive response time
Total Runtime
0
100
200
300
400
To
tal
Ru
nti
me
(sec
on
ds)
300 MHz 800 MHz 1.2 GHz 1.6 GHz 2.0 GHz
Median Response Time
050
100150200250300350400450500550600650
Res
po
nse
Tim
e (m
illi
seco
nd
s)
300 MHz 800 MHz 1.2 GHz 1.6 GHz 2.0 GHz
Other experiments
• Paper has details on more experiments
– Microsoft Windows, Linux
– Evaluate interactive performance of different disk IO schedulers
Experience using VNCplay
• Platform independence– Measured interactive performance of
PowerPoint, Word, OpenOffice
• Reliable replay– Replayed 5-minute PowerPoint session on a
slow system, taking over an hour – works OK!
Caveats
• Near-perfect reproducibility needed
– VMware: start the same VM every time
– In Linux, reset user account (dotfiles)
Caveats
• Keyboard input can be a problem– Use mouse instead of keyboard shortcuts– Click before and after you type
• Couple of tries to get a reliable session– Tooltips, pop-ups get in the way
Using VNCplay
• vncplay record server:5901 trace.vnc
• vncplay play server1:5901 trace.vnc out1.rfb• vncplay play server2:5901 trace.vnc out2.rfb
• vncplay analyze out1.rfb out2.rfb > analyze.out
• vncanalyze median analyze.out• vncanalyze cdf analyze.out > cdf.plot
Use gnuplot, Excel to plot cdf.plot
Other uses
• Automated testing– Make sure your application behaves the same
way with a new release of Linux / Windows
• GUI ``automation’’– Automate GUI-intensive tasks that have no
scriptable interface
Future directions
• Integrate some GUI toolkit knowledge for smarter matching (e.g. ignore tooltips)
• Better keyboard input handling– Machine learning– Details in paper
• If you want to help, let us know!
In conclusion …
• Need to measure interactive performance– Runtime benchmarks are not appropriate
• VNCplay– Portable, reliable session replay– Response time measurement
Questions?
http://suif.stanford.edu/vncplay/