Digital Pen and Paper Practices in Observational Researchubicomp.ucsd.edu/wp-content/papercite-data/pdf/weibel2012digital.pdf · Digital Pen and Paper Practices in Observational Research

Digital Pen and Paper Practices in Observational Research

Nadir Weibel, Adam Fouse, Colleen Emmenegger, Whitney Friedman, Ed Hutchins, Jim HollanDistributed Cognition and Human-Computer Interaction Lab

Department of Cognitive Science, University of California, San Diego9500 Gilman Dr., La Jolla, CA 92093-0515

{weibel, afouse, cemmenegger, wfriedman, ehutchins, hollan}@ucsd.edu

ABSTRACTResearchers from many disciplines are taking advantage ofincreasingly inexpensive digital video to capture extensiverecords of human activity in real-world settings. The abilityto record and share such data has created a critical moment inthe practice and scope of behavioral research. While recentwork [3, 5] is beginning to develop techniques for visualizingand interacting with integrated multimodal information col-lected during field research, navigating and analyzing theselarge datasets remains challenging and tools are especiallyneeded to support the early stages of data exploration.

In this paper we describe digital pen and paper prac-tices in observational research and their integration withChronoViz [4], a tool for annotating, visualizing, and analyz-ing multimodal data. The goal is to better support researchersboth in the field, while collecting data, and later in the lab,during analysis. We document the co-evolution of notetakingpractices and system features as 28 participants used the toolduring an 18-month deployment.

Author KeywordsDigital Ethnography; Paper-Digital Notes; Video Analysis;Interactive Navigation; Activity Visualization; Annotations

ACM Classification KeywordsH.1.2 User/Machine Systems: Human Factors;H.5.3 Group and Organization Interfaces: Computer-supported cooperative work, Evaluation/methodology, The-ory and methods

General TermsDesign; Experimentation; Human Factors

Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwise, orrepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee.CHI’12, May 5–10, 2012, Austin, Texas, USA.Copyright 2012 ACM 978-1-4503-1015-4/12/05...$10.00.

INTRODUCTIONNew digital devices and storage facilities are revolutionizingobservational research and are now common components ofdata collection. Multiple HD digital video and still cameras,directional audio microphones, depth cameras, wearable eye-tracking devices, GPS or other mobile sensors, and digitalpens are now typical tools for researchers in many disciplines.Data collection is being extended into real-world settings thathave not typically been accessible. Just as availablity of au-dio tape recording supported the development of conversationanalysis [14] and the ethnography of speaking [8], the ad-vent of a range of inexpensive digital recording devices andsensors promises to have a fundamental impact on a broadrange of scientific and engineering activities. The increas-ing availablity of this rich new data is important because tounderstand the dynamics of human and social activity, wemust also understand the full context of those activities andthis can only be accomplished by recording and analyzingdata of real-world behavior. While such data are certainlyneeded, more data cannot be the whole answer, since manyresearchers already feel that they are drowning in data.

New computational algorithms promise to aid analysis buttypically are available only after data have been selected, tran-scribed, and coded. For example, initial dataset navigation istypically cumbersome. When navigation and data selectionare costly, data records chosen for analysis are often a hap-hazard sampling of the recorded data, few analyses are done,and researchers come to have a large investment in selecteddata segments. Each analysis may appear as an isolated casestudy and it can be difficult to know how common the ob-served phenomena may be. Larger patterns and contradictorycases can easily go unnoticed, potentially compromising thequality of science.

One focus of our work is therefore on developing and as-sembling tools and practices to speed and improve the earlystages of annotation, navigation, visualization, and analysis.Raw data are usually coded and transcribed in a wide va-riety of ways, and coordinating multiple data sources andre-representations of the original events [7] is a challengingproblem that typically requires manual alignment and man-agement of the different data sources. Frequently researchersdo not have the time or the necessary resources for this task.

Session: The Tools of the Trade CHI 2012, May 5–10, 2012, Austin, Texas, USA

1331

Recent work [3, 4] addresses this issue by providing a tool,called ChronoViz, for interactive annotation, navigation, andvisualization of multiple data streams to help researchersmaintain a consolidated view of the whole data collection.

Even with better support for the management and integrationof multimodal data, identifying important moments deservingof detailed analysis remains challenging. This often requiresmultiple views (and viewings) of the data, integration withfield notes and insights from the researchers’ experience, anddiscussion with other analysts. We think it is particularly im-portant to facilitate these early stages of analysis and to doit in ways that integrate with current practices. For example,many researchers still rely heavily on paper-based documentsand hand-written notes during analysis. This is largely be-cause of the many affordances pen and paper still provideeven in today’s highly digitized work environments [15]. Pa-per often represents the only viable way to support the flex-ibility needed while collecting data and also continues to bean important resource throughout the analysis process.

We present an analysis of an 18-month deployment of a sys-tem for capturing paper-based notes in the field and its useby 28 researchers from multiple disciplines. We documenthow paper-digital tools influenced researchers’ practice andshaped their analysis of multimodal data. This in turn in-formed design of new paper-digital capabilities and new nav-igation and visualization facilities for ChronoViz [19, 4].

PAPER-DIGITAL NOTES IN THE FIELDIn order to support current notetaking practices, we devel-oped an integrated paper-digital workflow that allows fieldresearchers to use Livescribe digital pens1 to take notes. Thesystem employs ethnographer++ [19], a special softwarepackage installed on the digital pen that captures researchers’paper notes and automatically timestamps each pen stroke.The pen looks and functions as a normal pen but includes aninfrared camera and a processor that captures and decodes aspecial dot-pattern2 printed on paper to track pen tip move-ments in real-time. Livescribe pens and ethnographer++

work with any commercially available Livescribe notebookor with patterned paper printed with a standard color laserprinter.

After taking notes the pen can be docked to a computer todownload the notes. ChronoViz [4] exploits the recordedtimestamps to integrate and align the downloaded notes withother data collected during the observation session (e.g. mul-tiple videos, GPS data, still images, audio data, eye-tracking,etc.). Figure 1 shows an example of paper-based notes andrecorded video collected during a flight-simulation session.A separate window (Fig. 1, right) shows the collected notes.Interacting with the time slider or playing the ChronoVizsession will animate the notes by dynamically changing thenote color from semi-transparent (notes taken after the cur-rent time point) to opaque (notes taken before the current timepoint). Additionally, when researchers select a time point inthe video, notes taken at that time are highlighted. Similarly,

1http://www.livescribe.com2http://www.anoto.com

Figure 1. ChronoViz interface to access paper-based notes. On the left,a video frame of the observed flight simulation and annotations frompaper notes. On the right, the digital representation of the paper notesin ChronoViz, and a digital pen used to access the notes from paper.

users can click on a particular note on the digital represen-tation and the video will be positioned at the time that notewas taken. While Livescribe pens can be used to take notesin the field, these digital pens do not support wireless real-time interaction with a computer. To directly interact withChronoViz from the paper notes, Anoto DP-201 streamingpens can be connected to the computer over bluetooth andthe paper notes can be used in the same way as their digitalrepresentation on screen to navigate the data (Fig. 1, right).More detail about the integrated paper-digital workflow canbe found in [19].

Research DomainsThis basic infrastructure for paper-digital notetaking wasmade available for 18 months (starting in February 2010) to28 researchers in HCI, computer science, sociology, aviationhuman factors, cognitive science and animal cognition. Ineach case, we were participant observers. The research goalwas to observe how researchers used the tools and how theirintegration modified their current work practices. We ob-served the use of ChronoViz, and in particular its digital notescapabilities, during the entire deployment of the system. Ad-ditionally, we conducted regular focus group discussions andbrainstorming about the digital pen and paper capabilities ofthe system with researchers in all analyzed domains. We usedgrounded theory and performed affinity analysis to categorizeboth observational data and the collected material, and to ana-lyze trends and patterns evolving from the different practices.In this section we describe three research domains. In the nextsection we analyze usage and describe the resulting practices.

Aviation Human FactorsA team of ten researchers studying the dynamics of activity inthe cockpit of commercial airplanes participated in our study.The team was composed of researchers in the fields of avia-tion human factors, HCI, cognitive science, information sys-tems, information visualization, and computer science. Wealways had at least two researchers observing the use of thepaper-digital tools. Researchers captured and analyzed multi-ple data streams (behavioral notes, videos, photos, simulatorlogs, GPS, audio, eye-tracking, Apple iPad recordings, andMS Kinect depth cameras) during simulated flights on desk-top, fixed-based, and full-motion simulators.


1332

Figure 2. Aviation human factors research with ChronoViz on data collected at an airline training center. On the left, several video frames andannotations. On the right, a view of the paper-digital notes, audio transcript and the GPS track of the simulated flight.

Prior to introducing the paper-digital infrastructure, papernotebooks were commonly used to record pilot behavior overmultiple observation sessions. The researchers were providedwith our paper-digital technology and the ChronoViz tool.Figure 2 illustrates a typical analysis session with ChronoVizand the paper-digital notes. As described below, the researchteam experimented with a variety of notetaking strategies.

Animal CognitionWe also deployed our technology with a group of 4 re-searchers (one participant observer) studying animal cog-nition. This group investigates the activities of dolphins,chimps, and elephants. During the deployment they fo-cused on elephants and dolphins, observing interactions be-tween six adult female African elephants in captivity at theSan Diego Zoo Safari Park as they competed over shared foodresources and alliance formation during resource competitionamong bottlenose dolphins in Shark Bay, Western Australia.

Researchers collected video data as well as paper-based noteson ethograms, structured forms used by ethologists to recordparticular behavior or events of interest. With the introductionof the digital pens and ChronoViz, the researchers explored anumber of novel note-taking practices, including methods tosupport trajectory recording. Figure 3 illustrates a ChronoVizview of a video segment of elephants and the correspondingethogram record of the animals’ behavior.

Emergency ResponseFinally, we worked with twelve researchers in sociology, cog-nitive science, and computer science involved in an NIH-funded project on Wireless Internet Information System forMedical Response in Disasters (WIISARD3). The WIISARDresearchers (one participant observer joined this effort) cre-ated a set of paper-digital interactive forms in the months

leading up to an emergency response drill in July 2010. Theycollected multiple data points, including responder activities,locations, victim status and location, as well as position ofemergency vehicles. We analyzed the production of theseforms, how they were used during the exercise, and later dur-ing analysis of the collected data with ChronoViz.

ANALYSIS OF NOTETAKING PRACTICESFor the 18-month deployment, researchers were given digi-tal pens and access to patterned paper. They were instructedin basic use of the pens (e.g., starting, stopping, docking, anddownloading data from them), as well as current and potentialcapability of the paper-digital tools. While we actively partic-ipated in many data collection sessions, we also gathered theraw pen data and the paper-digital notes that researchers pro-duced. In the rest of this section we describe the structure ofnotes, use of symbols and gestures, and how the paper-digitalfacilities influenced notetaking strategies.3http://wiisardsage.ucsd.edu

Figure 3. Animal cognition research on elephants’ data collected at theSan Diego Zoo Safari Park. On the right, a particular digital ethogramused by the researchers.


1333

Figure 4. Notes evolution during the aviation human factors research. Left: a page from a user’s first session using a digital pen; notes are verbose andstill include time stamps. Center: more structured notes taken by a more experienced user; symbols and gestures start to appear, no time informationis noted. Right: structured notes taken by an expert user; gestures have been moved to a specific region, symbols are used to group notes.

Structure of the paper-digital notesPaper notes taken during behavioral research sessions areoften organized to reliably capture particular events at thetime they occur. When exact temporal data are required re-searchers may indicate a particular time as an event occurs.When these data are not critical to the planned analysis, re-searchers may structure notetaking into interval (such as one-minute) sampling periods. These notes often evolve overtime: researchers may begin a study by observing an event ofinterest and taking notes ad libitum, often taking advantage oflined paper to provide a temporal structure for the event. Theresult of these efforts is often a stack of pages with top-to-bottom inscriptions of selected aspects of observed activity,often including structured tables and diagrams. To use theinformation captured as data, researchers would normally re-turn to the lab to transcribe these notes into a digital database.

With the introduction of paper-digital tools, we observed aninteresting evolution of notetaking practices: certain hall-marks of observational research notetaking disappeared andresearchers began explicitly tagging their written notes forlater analysis. Though still written with pen and paper, thesenotes became live digital documents that no longer requiredthe extra step of transcription. Space and shorthand notationwere used in new ways to indicate meaning relative to the ob-servation session as well as to the way those data were parsedin the paper-digital workflow.

Figure 4 depicts an interesting example. Automatic time-stamping facilities of the digital pens freed researchers fromhaving to notate time (still present in Fig. 4, left), allowingthem to focus on what was going on rather than when it washappening. In this example, notes evolved to include hybridsolutions, containing a mix of linear notes and more symbolicrepresentations and special gestures. Additionally, particu-lar areas of the page (such as the margins) began to containspecific types of information. This trend emerged even moreclearly when researchers in aviation human factors began ex-perimenting with different spatial layouts for capturing activ-ity during flight. Figure 5 shows two examples.

Although the animal cognition researchers already used etho-grams in their notetaking, introduction of the digital pensystem changed their practice in interesting ways. As with

the evolution of notetaking practices by human factors re-searchers (Fig. 4), the animal cognition researchers alsostopped manually writing the time of each observation. Thefinal digital ethogram is a structured representation that al-lowed the researchers to quickly encode multiple details ofthe observed activity (Fig. 6, bottom). As we will see in latersections, researchers developed strategies for enabling subse-quent navigation and scoring of the collected data as part ofthe real-time note taking activity; once the pen data is trans-ferred to a computer, interpreting symbols and handwritingautomatically creates summaries and visualizations of the ob-served activity.

In the groups described above, use of digital pens was charac-terized by an iterative restructuring of their notetaking prac-tices. Aviation human factors researchers shifted from lin-ear plain notes to more structured notes and basic ethograms.Animal cognition researchers created complex digital etho-grams. Researchers using the digital pens for the emergency

Figure 5. Notes of cockpit activity. Top: the simulator’s Mode Con-trol Panel (MCP) annotated during flight simulation when controls werehandled by the pilots. Bottom: a hand-drawn diagram of the pilot’sinteractions with the flight instruments (throttle, flaps, yoke).


1334

response drill developed and evolved a set of 12 differentforms to capture the dynamics and locations of emergencyresponders, victims, and vehicles involved in the drill. Fig-ure 7 shows examples of three forms used during the drill.The structured forms were composed of standard elementssuch as checkmarks or text areas, but also included specialareas tailored for use with the digital pens, such as countingthe number of strokes to identify occurrences of a particularevent (see Fig. 7, bottom form).

Symbols and gesturesPaper-digital notes created in the different domains also in-volved distinctive symbols and gestures that participants in-troduced to augment notetaking. Symbols were used in bothaviation human factors and animal cognition domains to pro-vide a form of bookmarking. Some researchers used asterisks,others stars, and others simple shapes or checkmarks to book-mark a key moment in time as recorded in the timestamp ofthe pen stroke. Important benefits of these bookmarks were toaid future navigation through the collected data and highlightparticular events that researchers might want to return to dur-ing subsequent analyses. More complex and structured sym-bols were introduced later. For example, Figure 4 (center andright) shows use of a triplet of symbols (A-E-R) sometimesconnected to each other with a vertical arrow or a line. Avi-ation human factors researchers designed this coding schemeto capture pilot/copilot coordination in a cockpit: “A” encodesa request for action, “E” execution of the requested action,and “R” verbal response.

Arrows were frequently used by researchers in all domains.This is not surprising given this is a common practice in in-formal notes. However, it is interesting to examine the mean-ings of different arrows and how digital pens influenced us-age. Figure 4 depicts several types of arrows: the notes inthe center use arrows as bookmarks, to point out importantevents (‘→ 335 Hdg’ and ‘→ 065’), to encode direction andmovement (the ↗ after ‘brake release’ symbolizes take off,and the angled arrow just after encodes a turn). Notes on

Figure 6. Evolution of a digital ethogram for elephants’ interactions.Top: initial idea, still bound to time-coding the note. Bottom: Fullyfunctional digital ethogram encoding details about interactions, prox-imity and behavior of the different elephants (Ma, Mu, Mo, etc.).

the right-hand side of the same figure show an example of anarrow (←) used to emphasize the connection of a classifier(‘calm’) with a specific part of the group of notes. A simi-lar usage also appears in Figure 5 (bottom), where directionalarrows are used to represent the movement of controls as aresult of pilots’ actions at different times during a flight sim-ulation.

Figure 8 (left) depicts another usage of arrows. A re-searcher was trying to understand pilot/copilot communica-tion during a checklist procedure and used arrows to infercausality between a question and answer. But perhaps themost interesting use of arrows can be seen in Figure 8 (right),where researchers looking at dolphin alliance dynamics usedarrows as vectors to specify movement, direction, and loca-tion of different dolphins at different times relative to eachother. Finally, arrows have also been used to encode explicittrajectories on a map, as in the case of the elephants’ move-ments shown in Figure 9 (left).

The meaning of some symbols drawn with digital pens goesbeyond their visual appearance and involves the time taken todraw them. These symbols usually consist of a single strokeand represent the time course of an event by the time takento draw the symbol. For example, the continuous circles visi-ble in Figure 9 (right), indicate the time a specific coded task(executing the takeoff checklist) took to complete. A simi-lar coding is represented by the ⊥ symbol visible on the leftinsert in Figure 8.

Figure 7. Forms used during the WIISARD drill. Left: list of respondersactive in the field, the extra column has been added during the drill bythe researcher. Top-Right: medics work during triage/treatment in the‘Hot Zone’, negative numbers by the checkmarks are time corrections(in seconds). Bottom-Right: team leaders’ activity, the same activity canbe executed multiple times (multi-stroke cells in the center of the form).


1335

Figure 8. Two different uses of arrows. Left: observer looking at pi-lot/copilot communication during the execution of a checklist: notes arearranged vertically in chunks depending on the pilot/copilot coordina-tion and arrows indicate causality. Right: tracking dolphin alliance dy-namics while observing multiple synchronized videos in ChronoViz.

Paper vs. digital vs. paper-digitalTraditional paper notes are static representations often usedto aid recall and support analysis. One of their virtues is theflexibility to be tailored to specific representational needs andcontexts. This is no doubt part of why they are so commonin current practice. The introduction of digital pens makes itpossible to add many new facilities. We have seen how somefeatures evolved in the domains described above. The designspace of possibilities is complex, potentially domain specific,and involves many tradeoffs. With both traditional and digitalnotes some information is only implicitly represented, avail-able only as a result of interpretation by a person or a digitalsystem. How to design a system to benefit people and exploitcomputation presents numerous challenges.

Keeping all recorded information accessible is one challenge.In addition, deciding what information should be availabledirectly on paper, only within the digital system, or with bothinvolves complex tradeoffs. We observed various strategiesto cope with these tradeoffs. The localization of symbols andgestures in special areas of the page (e.g. Fig. 4, right), isone strategy. Symbols specifically meant to be interpreted areseparated from the rest of the notes, making it easier for thedigital processing system to recognize them, but also makingit easier for the user reviewing the paper notes to ignore themif they have no meaning as paper-only notes.

More structured notes and ethograms involved similar tech-niques. However, it is interesting and important to note thatwhile many notes are meant to be interpreted only by thecomputer, they were still represented by recognizable traceson paper. For example, in most forms from the emergency re-sponder drill (Fig. 7), there is no need to have a checkmark ora line on paper. A simple dot (even with a non-marking pentip) would be sufficient for the digital pen to record an entryand for the computer to process it, but having a record on pa-per turned out to be useful in practice. We also saw this inother situations such as when annotating ethograms or fillingout forms (Fig. 5, Fig. 6 bottom, and Fig. 9 right).

In other situations, specific symbols and gestures are explic-itly meant to be usable both on paper and later during Chron-Viz analysis. For example the A-E-R triplet (Fig. 4, centerand right) was used by a researcher during on-going observa-tions to keep track of pilot/copilot interactions. This notation

Figure 9. Left: tracking elephants movements through trajectoriesdrawn on a map. Right: aviation human factors researchers capturedpilot behavior, the letters represent coded sets of a pilot’s activities.

also provided the data for the visualization in ChronoViz ofthe density of interaction through time. Similarly, during theemergency drill, information written on the forms (Fig. 7)was used during data collection to double check if a particularevent was already captured. Other marks left on paper wereused to guide the collection of supplementary data. For ex-ample, researchers noting interactions during the drill neededto have paper records to cross-reference responders, victims,and vehicles across forms. As notes are taken, they establishthe context for subsequent note taking. For example, animalcognition researchers required proximity information to codeinter-elephant behavior (Fig. 6, bottom), and the vector in-formation of dolphin positions evolved as additional vectorswere sketched in positions relative to earlier sketch vectors(Fig. 8, right).

As we pointed out previously, not all information is actuallyavailable on paper and the only way to access it is to down-load the data and view it on a computer. For example, inorder to know the exact time or duration of a particular activ-ity tracked by continuous strokes (e.g. the circles in Fig. 9,right), the exact distance at any given time between the dol-phins tracked in the vector diagram (Fig. 8, right), or the timewhen pilots were interacting with specific instruments in thecockpit (Fig. 5), one needs to process the digital data.

Qualitative and quantitative data collectionIn most of the examples we observed, the digital pen sys-tem was used to collect both qualitative and quantitative data.Notes, typically short sentences or single words, usually sup-port qualitative analysis, while symbols and gestures wereused more commonly in quantitative analysis. For example,researchers using forms during the emergency drill (Fig. 7),introduced a number of ways to code data using single strokesfor binary data (e.g., strike marks visible in Fig. 7, top-right),multiple strokes for counting events (Fig. 7, bottom-right),and handwriting recognition for categorical data. We ob-served similar examples in the other domains: animal cogni-tion researchers marked particular spaces on their ethogramsto quantify the proximity of elephants (Fig. 6, bottom) andthe position of the dolphins via relative spacing of marks(Fig. 8, right). Aviation human factors researchers used sym-bols to quantify the number of A-E-R interactions (Fig. 4,center and right) and the specific activities performed by pi-lots (Fig. 9, right).


1336

Researchers also exploited the digital pen’s timestamping ofevery stroke in different ways. For example, this enabled datacollected on paper-digital forms for the emergency drill to besynchronized with the rest of the data, allowing researchers toknow when specific activities occurred, such as when a cer-tain victim was triaged (Fig. 7, top-right), and the exact timeof radio commands of specific individuals (Fig. 7, bottom-right). Animal cognition researchers used pen data to quan-tify occurrences of selected elephant behaviors and access theexact time of particular events (Fig. 6).

One interesting observation involved a tradeoff between re-cording qualitative and quantitative data. In fast paced envi-ronments like the cockpit of an airplane, spending time writ-ing details about one event means potentially not being able tonote other activities happening at the same time. One nice ex-ample of this tradeoff occurred when two researchers exper-imented with strategies for collecting data. Both researchersobserved the same event and used the same ethogram to col-lect data about activity in the cockpit. One strategy stressedthe qualitative nature of the notes, by inserting details aboutthe observed activity directly at the time it happened, whilethe other strategy attempted to capture the number of occur-rences and the time of the specific activity with more preci-sion using checkmarks.

Tension between time and spacePaper notes often exploit spatial organization. This addedstructure is useful both during recording and later during re-view. Paper-digital notes also introduce time as an additionalstructuring factor. Every single stroke is timestamped, en-abling researchers to easily access the temporal dynamics ofnote creation.

In all domains we studied, researchers mixed spatial and time-related structuring. Figure 4 (right) provides an example,where the notetaker grouped a set of earlier written notes withcurly brackets. In this case, the time when the curly bracketwas written is not important; position on the page carries itsmeaning. In Fig. 5, position of the notes on the ethograms isa key aspect of the underlying coding scheme, assigning it toa semantic category, but the time when it occurred is impor-tant too, reflecting when the pilot handled particular controls.This is particularly evident in structured ethograms, tables orforms (e.g. Fig. 6, Fig. 7). In some situations, not only theposition, but also the specific characteristics of a note are im-portant. In the example of the use of vectors to track dolphinsalliances (Fig. 8, right), the relative position of the arrows aswell as the numerical and textual attributes were required forcorrect interpretation of the event.

Finally, the time of the first interaction with a particular sec-tion of paper-digital notes or ethograms often reflects the timeof an event but subsequent interactions with the region onlyfurther elaborate the event and their time is not important.However, in other situations, such as in the multi-stroke fieldintroduced in some emergency drill forms (Fig. 7, bottom-right), every stroke added to the same field carries importanttime information. Deciding when time, space or both are im-portant in a paper-digital form is complex, requiring analy-sis of the specific forms and practices, and flexibility across

contexts. The advantage of using a digital pen is that bothlocation and time are always recorded and thus available forsubsequent analysis.

LimitationsThe long-term deployment of paper-digital tools demonstra-ted that researchers were successful in using them. In gen-eral the robustness of paper allowed researchers to overcomemost problems that occurred. Even when digital pens fail andall recorded data are lost, the paper documents still containvaluable information. For example, even if the timing of eachstroke in the elephant ethogram (Fig. 6) had been lost, the rel-ative timing of the events would have been preserved withineach minute of observation due to the structure of the etho-gram. It is worth noting though that lost data might havebeen recorded in another way (e.g., specific times written onthe forms) had participants not been using digital pens anddepending on them to capture specific data.

Other minor problems can occur even if a pen is working.For example, for structured digital notes in which locationon the page is important, a line or symbol that crosses fromone zone on the page to another can be a problem. Figure 10(left) shows a strike mark that may be counted twice becauseit appears in two distinct zones. An error correction heuristic,such as assigning a stroke to the zone that contains the clearmajority of stroke points, could be applied. Data can also belost if strokes are written outside the printed dot pattern area(Fig. 10, right). These problems can be expected to be moreprevalent when notetakers are hurried and less precise in penusage, as in the fast moving and chaotic disaster relief drill.Here we saw problems that arose because participants forgotto turn pens on after turning them off for a break.

Figure 10. Two examples of a data capturing issue. Left: the strikemark could be counted as being part of both the second and the thirdrow. Right: data written on the margin, where no pattern is printed,will not be recorded by the pen.

INTERACTIVE NAVIGATION AND VISUALIZATIONIn this section we discuss how the introduction and evolutionof paper-digital practices shaped development of digital toolsfor navigation and analysis of video data, interpretation ofspatial notes, gesture and character recognition, and manage-ment of notes.

Support for navigation and analysis with the penObservations of interaction with paper notes in ChronoVizshowed that it can be cumbersome to move between pen-paper and mouse-keyboard interfaces. To make such tran-sitions less frequent, we developed a control panel withcommon video controls (play, stop, fast-forward, rewind)printed on paper and used with Anoto streaming pens(shown in Fig. 11). This permits researchers to control thevideo without needing to tap on a specific note. For example,due to the inherent delay between observing an action andnoting it on paper, researchers will often need to position the


1337

video a few seconds before a note was written to better un-derstand the context of the activity, and then step through andobserve the details by moving frame-by-frame. Our paper-based control panel provides this and other facilities.

A common pattern in video analysis is to watch a segmentmultiple times, repeatedly moving back to a single point invideo. We support this practice with a tap-and-hold gesture.When a researcher holds the pen down on a note, the videowill start playing from the time point of the note and con-tinue playing as long as the pen is held down. To watch thatsegment of video again, the researcher can simply lift the penand then tap-and-hold in the same location to replay the videofrom the same point.

Another reason to move between pen-paper and mouse-keyboard interfaces is to record analysis notes. Sometimesthe keyboard provides a better interface, while at other timesthe pen is superior. We support flexible use of either entrymethod, with on-line notes entered with a streaming pen orkeyboard. New notes are associated with a point in time andcan be used in the same manner as other notes, including tap-ping on them to move back to the associated time point.

Interpretation of spatial notesAs we observed researchers developing new methods for us-ing the spatial position of notes and taking advantage of theflexibility offered by automatic capture of the time whennotes were written, we realized more sophisticated on-screencapabilities were also needed. First, we created ways to digi-tally associate forms printed on Anoto-pattern paper with thenotes recorded on those forms. Initial versions of the paper-digital tools required researchers to print forms on sheetsof pre-printed Anoto-pattern paper. While this was quickand convenient, the notes lost meaning when brought intoChronoViz without being associated with a specific form. Forexample, to understand what a check mark means, one needsto know the meaning of the box or column where the markwas made.

A second type of support involves the timing of groups ofmarks. For example, some researchers created areas wherethey could simply put tally marks every time a particulartype of activity occurred. To understand the timing of these

Figure 11. Support for multiple note taking sessions, with two existingsets of notes (green and orange) and a current set of notes (blue). Theon-screen representation (left) matches the printed notes and blue ink ofthe pen. At the bottom right is the printed video playback control panel.

Figure 12. Spatial selection of notes, showing the times for the selectednotes on the a timeline below the notes.

collections of marks, we created a way for the researchers todraw a box to select the region of the notes. The times ofthose notes are then shown on an interactive timeline, as de-picted in Figure 12, allowing the researcher to see the timecourse of particular types of activity as well as jump to spe-cific instances.

Gesture and character recognitionBy exploiting existing frameworks and toolkits, ad hoc recog-nition of symbols, handwriting and gestures (e.g. the A-E-Rsymbols of Fig. 4, or the data collected through the formsin Fig. 7) can be performed after downloading data from thedigital pens. We implemented dedicated recognition4 plug-ins that generate annotations for some of the symbols used byparticipants. These annotations are automatically aligned andmade available within the main ChronoViz interface.

Management of notesResearch participants requested the ability to add notes on topof existing notes and create customized forms that could beprinted multiple times. To support these requests we created asystem for managing digital notes. One facility it supports isthe ability to reprint existing notes. A new copy of a note setcan be printed and used for navigation and further notetaking.After multiple notetaking sessions, a researcher may want acopy of the notes with each session, or with individual ses-sions. These capabilities are now supported by ChronoViz,including support for printing the notes from each notetakingsession in different colors that are matched with the on-screendisplay, as shown in Figure 11.

DISCUSSIONThe work presented here was informed by earlier analyses ofuse of paper-digital notes in a variety of contexts. Mackay etal. [12] investigated how laboratory notebooks can be aug-mented by capturing hand-written information and linkingit to searchable electronic versions of the notebook. The4We experimented with MyScript ICR (http://visionobjects.com),the iGesture framework (http://www.igesture.org) and the $1 recog-nizer (http://depts.washington.edu/aimgroup/proj/dollar).


1338

seminal Audio Notebook [16] added audio recording to a nor-mal notebook, allowing paper-based notes to be automaticallylinked to speech recording. This work influenced the basicfunctionality that ethnographer++ and ChronoViz support.

More recently, two systems for capturing researchers’ paper-based notes have been developed. Yeh et al. developed asystem for field biologists enabling paper notes taken in thefield to be linked to other digital data such as sensor readings,digital photos, GPS logs, etc. [20]. Tabard and colleagues an-alyzed how biologists in a laboratory used a hybrid notebookwith information that spanned the electronic and the physicalworlds [17]. While the system we developed addresses simi-lar issues and in fact allows heterogeneous data to be grouped,linked, presented and analyzed, our primary focus is to de-velop and explore facilities that allow researchers to exploitpaper-digital tools to facilitate the early stages of annotating,navigating, and visualizing multimodal observational data.

A number of recent studies examined paper-digital note prac-tices in contexts such as music [18] and engineering [13].Other work investigated the incidental nature of notetak-ing [9] and characterized natural interactions with digital penand paper [2]. While these studies describe the nature and theprocess of standard notetaking practices in different domains,the digital pen and paper notetaking system described herefocuses on the use of paper-digital tools to support behavioralresearch. We analyzed strategies and practices that evolved inusing our system in three settings.

The tools we introduced changed the way researchers col-lected data and ultimately affected the kind of data they col-lected. The researchers were able to leverage the portabilityand flexibility of paper [15], and the naturalness and spon-taneity of the notetaking process [1], while still exploiting thecomputational power of the digital pen and paper system.

We documented how researchers began using the hybridpaper-digital system much as they would use traditional penand paper, even continuing to manually timestamp their ob-servations (left image in Fig. 4 and top image in Fig. 6). Withmore experience they began introducing bookmarks, sym-bols, gestures, and increasing structure into their notes (centerimage in Fig. 4). As the researchers continued to use the penand paper system they began to see additional ways to exploitthe computational aspects of paper-digital notetaking technol-ogy (top image in Fig. 5, bottom image in Fig. 6, Fig. 8, andFig. 9). It is at this point that our participants seemed to fullyrealize that they were using a computer and not just a pen andbegan to modify their notetaking practices, thereby engagingin a paradigm described by Guimbretiere as cohabitation [6].

Paper notebooks are extremely flexible and personal. We ob-served, as previously discussed by Mackay et al. [11], howpaper allows notes to be spatially organized in ways rarelysupported as well by digital tools. It also permits later ad-dition and annotation. Figure 7 shows addition of notes inthe bottom margin as well as an entire column on the left.The ethograms in Figure 5 were both created after the re-searchers arrived at the data collection setting. The MCP dia-gram was printed on dot paper (top), and the yoke and throttlewere hand drawn (bottom) at the study site, just before data

collection sessions began. Notes can be augmented digitallyand reprinted. Paper can no longer be considered simply tem-porary storage as described by Lin et al. [10], but becomes apersistent tool used for both data collection and multiple cy-cles of analysis.

The introduction of paper-digital tools involves importanttradeoffs: timestamped notes vs. spatial arrangement, qual-itative vs. quantitative analysis, and paper-only vs. paper-digital access to the information. We discussed the impor-tance of the physical layout of notes [11] and how researchersoften used a combination of time-based notes and spatialstructuring and linking. The ethograms of Figure 5, Figure 8and Figure 9 (right), and the trajectories drawn in Figure 9(left) are examples. In the analysis section we outlined howresearchers made use of the paper representation during ob-servation, but also counted on automatic capture of times-tamping and linking data in later use. The tension betweencapturing more information quickly and creating detailed re-ports (as outlined by Bernstein et al. [1]) was mitigated byautomatic capture rather than requiring coordinating a set ofrecording devices (video cameras, audio recorders, GPS andother sensors). Most of the time researchers did not have tothink about this issue. However, we did witness researchersstill considering this tradeoff and attempting to develop a so-lution that best suited the particular observation, employinga division of labor between notetakers, with each specificallylooking at one aspect of the observed phenomena.

As we learned about how notetaking practices were changedby the use of digital pens, we also implemented new ways ofintegrating pen-based actions into ChronoViz, such as facili-ties that permit users to use pen interactions to control videoplayback. The practice of associating the location of penstrokes with meanings led us to develop a way to define mean-ingful regions on pre-printed forms (including ethograms).Later versions allow specific pen gestures to spawn partic-ular annotations in ChronoViz and permit users to add newnotes on top of existing notes. Responding to the evolvinguse practices of our user community, the refined system betteraddresses the challenges raised in the introduction: managingmultiple data streams, navigating large data sets, and find-ing important moments in the data. Because these challengesmust be confronted in any observational research program,the evolving suite of tools described here should be widelyapplicable.

FUTURE WORKDeployment of our tools allowed us to examine notetakingpractices and practical issues in three field research domains.In the future we expect to continue to study usage in these ar-eas, exploiting the research relationships we have developed.We also expect to expand our studies. For example, discus-sion with flight deck designers and airline instructors indicatethat a number of airlines are in the process of restructuringtheir training programs. Since flight instructors routinely takenotes during simulated flights, with our tools the notes can becombined and linked to other data about pilot performance(simulator logs, videos, etc.). This can in turn support de-briefing sessions by aiding navigation of pilot performancedata and supporting discussion of it.


1339

We are also investigating how people exploit paper in othersettings. For example we are examining how paper-digitaltools can be used to document mathematicians’ constructionof proofs and visualize the dynamics of this process. We areworking on expansions of current work with animal cogni-tion researchers, especially to support the reconstruction ofanimal trajectories. A researcher watching a single videofrom a fixed viewing angle can use the paper-digital tools totrace a two-dimensional trajectory for each animal in view.Similar traces constructed from other viewing angles can becombined to create a composite 3D trajectory visualizationin ChronoViz; something the researchers cannot do from anysingle view point. We envision using paper notes to recon-struct trajectories of elephants and captive dolphins.

CONCLUSIONUntil we began exploring uses of digital notes, it was difficultto appreciate the extent to which the activity of notetakingwould be changed by the new technology. Our tools trans-form observer notetaking from an activity that generates yetanother sort of hard to analyze data into an activity that gen-erates a powerful interface to digital data of all types.

The possibility of recording data digitally during observationpromises to transform current notetaking and data collectionpractices. In this paper we presented and analyzed the evolu-tion of paper-digital techniques developed by researchers inthree different domains. The resulting techniques exploitedseveral affordances of the novel paper-digital notetaking sys-tem we are developing, demonstrating its potential to supportfield work. In the process we extended ChronoViz – a toolfor annotating, visualizing, navigating, and analyzing mul-timodal data – to aid and shape these emerging notetakingpractices.

New analysis procedures are made possible by the use ofdigital-pen-based tools, including adding analysis notes toobservation notes, visualizing the temporal relations amongnotes, and quantifying the occurrence of various classes ofnotes. We intend to continue development of tools for ob-servational analysis and anticipate many, as yet unexplored,practical applications of this technology.

ACKNOWLEDGMENTSWe thank Christine Johnson for working with us on thedigital ethograms and access to rich data on animal cogni-tion. We also thank David Kirsh for assistance in the WI-ISARD project, and Richard Connor for the use of dolphinvideo. Support for this research was provided by NSF award#0729013, and UCSD-Boeing project agreement 2011-012.

REFERENCES1. Bernstein, M., Van Kleek, M., Karger, D., and Schraefel,

M. Information scraps: How and why informationeludes our personal information management tools.ACM TOIS 26, 4 (2008), 1–46.

2. Brandl, P., Richter, C., and Haller, M. Nicebook:supporting natural note taking. In Proc. CHI ’10 (2010),599–608.

3. Fouse, A., and Hollan, J. D. DataPrism: A tool forvisualizing multimodal data. In Proc. MeasuringBehavior 2010 (2010), 1:1–1:4.

4. Fouse, A., Weibel, N., Hutchins, E., and Hollan, J. D.Chronoviz: a system for supporting navigation oftime-coded data. In Extended Abstracts of CHI 2011(2011), 299–304.

5. Goldman, R., Pea, R., Barron, B., and Derry, S., Eds.Video Research in the Learning Sciences. 2007.

6. Guimbretiere, F. Paper Augmented Digital Documents.In Proc. UIST ’03 (2003), 51–60.

7. Hutchins, E. Cognition in the wild. MIT Press, 1995.8. Hymes, D. The ethnography of speaking. Anthropology

and human behavior 13, 53 (1962), 11–74.9. Ispas, A., Signer, B., and Norrie, M. C. A study of

incidental notetaking to inform digital pen and papersolutions.

10. Lin, M., Lutters, W., and Kim, T. Understanding themicronote lifecycle: improving mobile support forinformal note taking. In Proc. CHI 2004 (2004),687–694.

11. Mackay, W., Fayard, A., Frobert, L., and Medini, L.Reinventing the familiar: exploring an augmented realitydesign space for air traffic control. In Proc. CHI ’98(1998), 558–565.

12. Mackay, W. E., Pothier, G., Letondal, C., Bøegh, K., andSørensen, H. E. The Missing Link: Augmenting BiologyLaboratory Notebooks. In Proc. UIST ’02 (2002),41–50.

13. McAlpine, H., Hicks, B., Huet, G., and Culley, S. Aninvestigation into the use and content of the engineer’slogbook. Design Studies 27, 4 (2006), 481–504.

14. Sacks, H. Lectures on conversation, vol. 1. 1995.15. Sellen, A., and Harper, R. The Myth of the Paperless

Office. The MIT Press, 2003.16. Stifelman, L., Arons, B., and Schmandt, C. The audio

notebook: paper and pen interaction with structuredspeech. In Proc. CHI ’01 (2001), 182–189.

17. Tabard, A., Mackay, W., and Eastmond, E. Fromindividual to collaborative: the evolution of prism, ahybrid laboratory notebook. In Proc. CSCW ’08 (2008),569–578.

18. Tsandilas, T., Letondal, C., and Mackay, W. Musink:composing music through augmented drawing. In Proc.CHI ’09 (2009), 819–828.

19. Weibel, N., Fouse, A., Hutchins, E., and Hollan, J. D.Supporting an integrated paper-digital workflow forobservational research. In Proc. IUI 2011 (2011),257–266.

20. Yeh, R. B., Liao, C., Klemmer, S. R., Guimbretiere, F.,Lee, B., Kakaradov, B., Stamberger, J., and Paepcke, A.ButterflyNet: A Mobile Capture and Access System forField Biology Research. In Proc. CHI ’06 (2006),571–580.


1340