Upload
brrian
View
1.852
Download
4
Tags:
Embed Size (px)
DESCRIPTION
These are the slides for a series of work-in-progress talks given about Timelapse in June 2012. Please send comments and questions to @brrian on twitter, or find my email on the Timelapse project page: http://cs.washington.edu/homes/burg/timelapse/
Citation preview
Timelapse Interactive record/replay for the web
Brian Burg Andrew J. Ko Michael D. Ernst
Computer Science and EngineeringUniversity of Washington
Reproducing program behavior is essential in software development
Demonstration:
interacting with a program to achieve a behavior
Investigation:
interacting with developer tools to understand behavior
The phases of demonstrating and investigating are inseparable.
Task coupling can be hazardous
Example: debugging a color picker widget
Barriers to reproducing behavior
Demonstration
• Tedious, time-consuming, and manual
• Difficult to explain repro. steps (e.g., in a bug report)
Investigation
• Repeated demonstration to investigate same behavior
• The behavior being investigated is not held constant
• Interleaving the two phases incurs task switching penalties
Simplifying by decoupling
The means of investigation could change independently without re-demonstration.
Decoupling behavior demonstration from investigation simplifies reproduction and enables new practices.
Behavior could be demonstrated, and later, investigated, by different people.
Behavior could be reproduced precisely, automatically, quickly, and easily.
http://dryicons.com, http://www.getfirebug.com/
Solution: deterministic record/replay
User-space librariesJockey (C), Mugshot (JavaScript)
Application-specificVideo game engines
Operating systemsdOS, Determinator
Virtual machinesReVirt, LEAP, VMWare Workstation
HardwareDMP, RCDC, Calvin
Old idea: achieve identical execution by controlling nondeterminism.
Recent idea: modern browsers are virtual machines.
New idea: adapt virtual machine record/replay techniques to web browsers
Timelapse: record/replay for the web
Contributions
• Adaptation of VM record/replay techniques to the web
• Infrastructure and tool for interactive record/replay
• Design concepts for visualizing, interacting with recordings
Timelapse is an infrastructure and developer tool to record, replay, and investigate web application behavior.
Demonstrating and Investigating(with Timelapse)
Example: color picker, revisited
1. Begin recording2. Demonstrate behavior3. End recording4. Investigate as needed
Recreating a specific execution
To eliminate nondeterminism,
• Identify what is/isn’t deterministic
• Decide what sources to capture, and how
• Figure out how to replay the recorded nondeterminism
Timelapse adapts definitions and techniques from VM research.
Executions differ because of sources of nondeterminism.
Programs and Inputs
main.js
An execution is determined by a program and its inputs.
main.js
function foo(a,b) { return a + b; }
function bar(c) { return c * Math.random(); }
foo(argv[1], argv[2]);
bar(argv[1]); bar(argv[3]);
main.js
Programs and Inputs
main.js
An execution is determined by a program and its inputs.
main.js
function foo(a,b) { return a + b; }
function bar(c) { return c * Math.random(); }
foo(argv[1], argv[2]);
bar(argv[1]); bar(argv[3]);
main.js
Programs and Inputs
Sources of nondeterminism that affect the course of execution are implicit inputs to the program.
function foo(a,b) { return a + b; }
function bar(c) { return c * Math.random(); }
foo(argv[1], argv[2]);
bar(argv[1]); bar(argv[3]);
An execution is determined by a program and its inputs.
main.js
Interpreters and Inputs
>> foo(a,b) { return a + b; }
>> bar(c) { return c *
Math.random(); }
defs.js
>> load(“defs.js”);
>> foo(1,2);
>> foo(3,4);
>> bar(5);
>> bar(5);
REPL session
1. load defs.js
2. CALL foo 1, 2
3. CALL foo 3, 4
4. CALL bar, 5
5. RET Math.random, 0.453863
6. CALL bar, 5
7. RET Math.random, 0.986364
interpreter inputs
The session could be reproduced exactly by reusing inputs.
In an interpreter session, the program is the interpreter itself.
Web browsers are interpreters
Web browsers interpret JavaScript, CSS, HTML, and user input.
Web Interpreter OutputInput
Web browsers are interpreters
Executions can be reproduced by capturing and reusing inputs.
Web Interpreter OutputInput
Inputs Log
Record/replay goals
Precise deterministic record/replay• Must capture and reuse explicit and implicit inputs• Able to detect divergence of replay from recording
Self-contained, sharable recordings• Must not depend on any machine-specific context• Replay does not require external communication
Integration with existing architecture and tools• Record/replay must have low performance overhead• Controllable and compatible with existing developer tools
Record/replay technique: proxying
During normal execution, Date.now() returns the current time.
>> foo(a,b) { return a + b; }
>bar(c) { return “now:”+
Date.now(); }
/* file: Source/wtf/DateMath.h */
inline double jsCurrentTime(){return floor(WTF::currentTimeMS());}
Record/replay technique: proxying
During recording, the return value of Date.now() is saved.
>> foo(a,b) { return a + b; }
>bar(c) { return “now:”+
Date.now(); }
/* file: Source/wtf/DateMath.h */
inline double jsCurrentTime(){return floor(WTF::currentTimeMS());}
Inputs log
Record/replay technique: proxying
On replay, the logged return value of Date.now() is used.
>> foo(a,b) { return a + b; }
>bar(c) { return “now:”+
Date.now(); }
/* file: Source/wtf/DateMath.h */
inline double jsCurrentTime(){return floor(WTF::currentTimeMS());}
Inputs log
Shim: the thing in the middle
Shims are used to implement deterministic record/replay.
The hard part of implementing record/replay is designing and placing shims.
Background: browser architectures
Web interpreters are embeddable and cross-platform.
User input
Callbacks, visual output
Date.now, setTimeou
t
timer fire
Embedders Web Interpreter(WebKit, Gecko)
Platforms
Shim placement in the browser stack
Shims are placed at the API boundaries of the web interpreter.
Embedders Web Interpreter(WebKit, Gecko)
Platforms
Comparison: shim placement
For record/replay, VM only needs shim between OS/hardware
Why not place shims elsewhere?
At the OS or VM level (VMWare Replay)
Low overhead and exact replay
Can’t integrate with developer tools
At the JS library level (Mugshot: Mickens et al., 2010)
Cross-platform
Slower, lower fidelity, can’t integrate with tools
At the level of DOM events only (WaRR: Adrica et al., 2011)
Most DOM events are deterministic and/or have no effect
Skipping pre-dispatch browser code causes strange behavior
Interpreter inputs by source
User: mouse, keyboard, scroll, resizeNetwork: images, scripts, HTML, AJAXCommands: navigate, open, close page
Internal nondeterminism:Animations, transitions, multimedia,async. actions (load, error, popstate)
Functions: Date.now, Math.randomCaching: resources, cookiesTimers: timer interrupt schedule
“Pull” vs. “Push” inputsPUSH
TOmouse, keyboard, scroll, navigate, network events
resource data,client API hooks, input handling
PULL FROM
Animation, transition, load, error, media events
timer callbacks timestamp, random, cookies, persistent state
interruptsRTDSC, etc
Comparison: “Pull” vs. “Push”
Mechanics of execution replay
Injecting “push” inputs causes dispatch of DOM event(s).
1. mousedown
2. mouseup
3. click
4. focus
5. submit
DOM Event DIspatch
function onClicked(..) {
... = Date.now();
}
JavaScript execution
Comparison: replay timing
Problem: when to inject “push” inputs to cause same execution
VMs and Timelapse share notions of counting and injecting.
• Count branches, instructions
• Inject hardware interrupts
VM record/replay
• Count DOM event dispatches
• Inject “push” inputs
Browser record/replay
Challenge: finding and taming nondeterminism
Taming nondeterminism is easier with principled design.
Networked games like Starcraft 2 rely on record/replay facilities.
Video games explicitly design to a record/replay-like architecture.
Taming nondeterminism with shims relies on well-defined APIs.
Taming nondeterminism post-hoc requires refactoring and hacks.
Challenge: dealing with real web platforms
http://hawanja.deviantart.com/
• >1 MLOC (mainly C++/ JavaScript)
• Hundreds of committers, and 500-1000 commits per week
• 6 independent build systems
• Supports many OS, CPU, GUI toolkits, graphics engines, toggleable features, embedders
Web interpreters are complex, many-limbed, and fast-changing.
Lesson: consider many, implement one
http://commons.wikimedia.org
It’s fine to choose one reasonable configuration. But, don’t ignore essential complexity.
Platform-specific examples:• input method editor handling
• rendering and animation
• resource caching, cookies
Embedder-specific examples:• JavaScript engine
• Embedding API usage
For research, only address essential architectural complexity.
Interacting with a recording
The Omniscient Debugger for JavaControlling video on YouTube
Interacting with Whyline [Ko & Myers]
Whyline for Java: program output Whyline for Java: program output
Interacting with Timelapse
Timelapse Panel inside Web Inspector Application being replayed
Timeline visualization
InfoVis mantra: overview, zoom and filter, details-on-demand.
Filter inputs by
type
Program inputs over
time, by type
Time ruler
Timeline visualization
InfoVis mantra: overview, zoom and filter, details-on-demand.
Filter inputs by time span
Table of program inputs
Input types by
color
Inputs by id and time
Table is adjustable
Global record
controls
Panel-specific controls
InfoVis mantra: overview, zoom and filter, details-on-demand.
Recording, replaying, and modality
• Recording can be started/stopped from any Inspector panel
• Replay is controlled from the Timelapse panel only
• Modality displayed next to record/replay controls
Slider control and modality• Timeline and table are linked by sliders and filters.
• Dragging slider seeks replay to the dropped position.
Sliders show same position Dragging induces drop previews
How do developers use Timelapse?
Actual use cases? Good affordances? Workflow integration?
Formative user study
• Subjects: 9+3 professional web developers
• Tasks: create a test case, fix a bug
• Time: 45 min/task, + tutorial & exit interview
• How: within-subjects design, Camtasia, Timelapse
Important use cases
Play, watch and pause
Visual binary search
Stepping through inputs
Execution anchoring
Refining repro. steps
Important use cases
Play, watch and pause
Visual binary search
Stepping through inputs
Execution anchoring
Refining repro. steps
Important use cases
Play, watch and pause
Visual binary search
Stepping through inputs
Execution anchoring
Refining repro. steps
Important use cases
Play, watch and pause
Visual binary search
Stepping through inputs
Execution anchoring
Refining repro. steps
Important use cases
Play, watch and pause
Visual binary search
Stepping through inputs
Execution anchoring
Refining repro. steps
Important use cases
Play, watch and pause
Visual binary search
Stepping through inputs
Execution anchoring
Refining repro. steps
Study Results
No significant effect on task completion time. Why?
• Timelapse didn’t affect types of information sought
• “Program inputs” not related to existing concepts
• New strategies only possible with tight integration
Interactive record/replay mainly* accelerates existing strategies.
Future Work
“Canned” executions can bring research ideas into practice.
Dynamic analysison-demand race detection, code coverage, optimizations
Program synthesiscreate automated tests from recorded interactions
Corpus analysistrain models, test type systems, or evaluate analyses
Future Work
“Canned” executions can bring research ideas into practice.
Dynamic analysison-demand race detection, code coverage, optimizations
Program synthesiscreate automated tests from recorded interactions
Corpus analysistrain models, test type systems, or evaluate analyses
On-demand dynamic analysis
Dynamic analyses generate data at runtime with instrumentation.
Instrumented execution
Generated data
Dynamic analysis
Analysis Output
On-demand dynamic analysis
Record/replay enables on-demand dynamic analysis tools.
Timelapserecording
Normal execution
RECORD
Instrumented execution
REPLAY
Developer tools
Data and analysis
Conclusion
Timelapse decouples demonstration and investigation using ideas from virtual machine record/replay work.
The Timelapse tool supports interactive inspection.
Timelapse’s infrastructure is useful for other purposes.
The Timelapse research prototype is an open-source fork of WebKit, and is freely available from the project website.
https://bitbucket.org/burg/timelapse/
CALL FOR BACKUP
Network: a real proxy, not a shim
This is a hack.
In the short-term, easier than attempting to recreate platform-dependent network streams/handles
Long-term solution:
A resource loader implementation with built-in record/replay primitives, or a higher-level network API
An HTTP proxy records and replays network traffic.
Replay fidelity and completeness
Web interpreters expose a large and ever-changing API.
Timelapse doesn’t tame all sources of nondeterminism.
Excepting untamed sources, the DOM tree and JavaScript heap are identical for all recorded and replayed executions.
Divergence is automatically detected by comparing DOM events.
Divergence detection supports piecewise implementation.
Performance characteristics
Record and replay slowdowns are negligible because shims are placed along existing architectural boundaries.
Timelapse is ideal for CPU- and user-bound programs.
Network-bound applications are slower, due to prototype shortcuts (disabling caching and pipelining).
Replay/seek speed could be improved dramatically by checkpointing and clipping unnecessary layout/renders.
Performance hasn’t been a priority, but is adequate anyway.
Embedding and platform APIs
Embedders
EMBE
DD
ING
API
PLAT
FORM
API
Web Interpreter(WebKit, Gecko)
Abstraction layers separate web interpreters from platforms/embedders.
Platforms
Embedding and platform APIs
Embedders
EMBE
DD
ING
API
PLAT
FORM
API
Web Interpreter(WebKit, Gecko)
Abstraction layers separate web interpreters from platforms/embedders.
Platforms
Embedding and platform APIs
Embedders Web Interpreter(WebKit, Gecko)
Platforms
EMBE
DD
ING
API
PLAT
FORM
API
Shims sit between the web interpreter and abstraction layers.
Embedding and platform APIs
Embedders Web Interpreter(WebKit, Gecko)
Platforms
EMBE
DD
ING
API
PLAT
FORM
API
Shims sit between the web interpreter and abstraction layers.
Shim behavior when recording
[user mouse press]-> (IPC communication)-> WebKit::WebPage::mouseEvent()-> WebCore::InputProxy::handleMousePress()---> WebCore::DeterminismController::capture()-> WebCore::EventHandler::handleMousePress()-> WebCore::Node::dispatchMouseEvent()
EMBE
DD
ING
API
PLAT
FORM
API
Start recording!
Shim behavior when recording
[user mouse press]-> (IPC communication)-> WebKit::WebPage::mouseEvent()-> WebCore::InputProxy::handleMousePress()---> WebCore::DeterminismController::capture()-> WebCore::EventHandler::handleMousePress()-> WebCore::Node::dispatchMouseEvent()
EMBE
DD
ING
API
PLAT
FORM
API
EMBE
DD
ING
API
PLAT
FORM
API
Shim behavior when replaying
DeterminismController::dispatchNextAction()-> MousePressAction::dispatch()-> InputProxy::handleMousePress(true)-> EventHandler::handleMousePress()-> Node::dispatchMouseEvent()-> WebKit::WebPage::mouseEvent()-> InputProxy::handleMousePress(false)
Start replaying!
EMBE
DD
ING
API
PLAT
FORM
API
Shim behavior when replaying
DeterminismController::dispatchNextAction()-> MousePressAction::dispatch()-> InputProxy::handleMousePress(true)-> EventHandler::handleMousePress()-> Node::dispatchMouseEvent()-> WebKit::WebPage::mouseEvent()-> InputProxy::handleMousePress(false)