The Design Cycle and Brainstorming

AutoComPasteAuto-Completing Text as an Alternative to Copy-PasteShengdong (Shen) Zhao 1 Fanny Cheviler 2 Wei Tsang Ooi 1 Chee Yuan Lee 1Arpit Agarwal 1,3

1

Good afternoon, my name is Shengdong Zhao. You can call me Shen. I am an assistant professor at the National University of Singapore, which is in an island in the south east of Asia. I started and manage the NUS-HCI Lab. This work is done in collaboration with Fanny Cheviler, who is currently a Post-doc researcher at OCAD University in Canada, and Ooi Wei Tsang, who is my colleague in Singapore, and Chee Yuan and Arpit, who are former students and interns in the lab.

So what is this work about:

(add affiliations for Fanny and Arpit)

Presentation flow: 1) Title slide: (using NUS-HCI as the logo) 2) Copy-paste is a common activity: 3) However, it is not necessarily optimized (show the current work flow of it) (Can we perform copy-paste without the tedium of window switching and highlighting?)

4) What about using AutoCompletion for copy-paste purposes?

5) but it has some challenges (one dictionary size, two, unit of text)

6) We present AutoComPaste

AutoCompaste is built on several assumptions (are these assumptions true)?

How does AutoComPaste compare with traditional copy paste in different scenarios?

To answer the first question, we carry out a Field study.

to answer the second question, we carry out a controlled experiment and a qualitative study.

Future work and limitation:

----Ok, the overll flow is ok Need to promote the 1Background & Motivation2

is a common computing operationit often happens across documents

as the title implies, its about some innovation we have done related to the copy-paste operation, which is very common computing operation that all of us probably perform multiple times everyday. Well, if you do if very often, you might get lucky, like the parents of this beautiful twins.

When we perform copy-paste, we often done it across different documents, in which we open many documents, and copying the various pieces of content from other documents to a working document.

Many of the copy-paste operations are done across multiple documents, in which we replicate a piece of content from one document to another. 2Background & MotivationCurrent copy-paste techniques: 3

Ctrl-C, Ctrl-VMenu selectionDrag & dropX-WinChapuis and Roussel. Copy-and-paste between overlapping windows. CHI 07There are a variety of ways the copy-paste operation can be carried out, in particular, 36-Step Common Workflow4

I know you cant read it from the back, so I will try to explain the 6 steps one-by-one. 46-Step Common Workflow5

Step 1: TypingThe first step, which is often a pre-step for copy and paste, is editing some document, in which you are typing some text. Then, you realize that you need to copy some content from another document56-Step Common Workflow6

Step 2: Context switch& Win manageIn this case, you need to switch your context, and navigate and find the window or document you believe the copied content is located, 66-Step Common Workflow7

Step 3: Visual searchThen, you have to perform a visual search to find the exact content you want to copy. 76-Step Common Workflow8

Step 4: Highlighting & CopyThen you have to use your mouse or some kind of pointing device to acquire and highlight the text, then perform the copy operation (whatever it is) 86-Step Common Workflow9

Step 5: Window managementThen navigate back to the original editing document or window96-Step Common Workflow10

Step 6: Paste

Then perform the paste operation. This workflow applies to all the 4 copy-paste operation we have mentioned earlier. 10

6-Step Common Workflow11However, when we exam these operations, we found it involves quite a bit of windows management, which can be unnecessary, also, step 3 and 4, visual search, acquire target and highlighting, might not be required either. Is there any ways that I can simplify this process, at least for some scenarios? 1112

Instead of go to the source document to acquire the text, what if we build an index of currently opened documents, and use AutoCompletion to fetch the desirable content to copy directly from the editing window or document? 1213

+ Text Unit AdjustmentsAuto-Completing Text as an Alternative to Copy-PasteSo we replace the middle 4 steps with typing some prefix of keyword of the text we want to copy, and perform selection from a list. But we have to be cautious here, since we dont know how much text the user wants to copy, so we present the user with a default option, such as the most likely phrase or sentence, and provide ability for users to adjust it after the selection is done. So this is the basic idea behind AutoComPaste. This may sound like a good idea, but there are a number of assumptions we need to check increase our confidence of it. 1314

+ Text Unit AdjustmentsWindow management is common and tediousCopy-paste often Interleaves typingCopy-paste different sizes of text is commonSince AutoComPaste mainly saves on the time for windows management and visual search, we need to ensure that such type of copy-paste is common, and hopefully it is somewhat tedious and problematic currently.

Second AutoComPaste is keyboard centric technique, it will makes more sense if users often interleaves typing and copy-paste operations, so we want to find out if copy-paste does interleave with typing often. Third, what is the typical content size users copy and paste from? How much do we need to support autocompletion of different unit of text? These assumptions or questions will affect the viability or design of AutoComPaste, so we decide to carry out a log study to find out more. 14Logger StudyLogger that logs copy-paste event Automatically turned on, data send to a central serverFor each copy-paste event, we recordType (copy | paste) Number of windows open, host window, and application nameTimestampNearest typing event in terms of timeContent copied joe12@gmail.com is stored as xxx00@xxxxx.xxx Participants22 students (9 female, 13 male, 21-27, M 23.14) Duration2 weeks

15We developed a logger that can be automatically turned on.

22 participants (9 female, 13 male, aged 21-27, mean 23.14) tookpart in the 2-week study. All are university students in ComputerScience or Computer Engineering. Each participant was rewarded1% course credit after completing the study.

We developed a logging mechanism that collects CP activitiesrunning on the Windows XP/Vista/7 OS. Participants were askedto install the logger on their primary computer for a period of 14days. The logger was automatically turned on without any extraoperation from the user, and therefore was constantly running onthe background. Logs were periodically sent to our server.For each CP event, the logger logs its type (copy or paste), thehost window and application, the timestamp, and the content copied.We also record the time difference to the nearest typing event whenit applies (duration between a CP event and the latest typing eventperformed before, and the earliest typing event performed after).For each text object copied, we log its content by masking alphabeticcharacters and numerical digits to protect the users privacy(e.g. joe12@gmail.com is stored as xxx00@xxxxx.xxx).Punctuation and whitespace are preserved to retain structural informationsuch as the number of words, sentences, and paragraphs.15Logger Study - ResultData collected34.1 MB of text data, 8168 events with 3481 (43%) copy and 4687 (57%) paste.Windows opened 83% of the time, users have 6-20 concurrently opened windows (average 12) when performing CPType of copy-paste57% (2672) cross-document CP 43% (2015) within-document CPInterleaving with typing42% of copy events were performed after typing, and 54% of paste events were followed by typingText size Phrases (39%), Sentences (33%), Paragraphs (28%)

16A total of 34.1 MB of text logs were collected. Among the 8168events, 3481 (43%) were copies and 4687 (57%) were pastes. Asimilar distribution was observed in [15].Windows management. We found that 83% of the time, usershave 6-20 concurrently opened windows (average 12) when performingCP. Moreover, among all the 4687 pastes, cross-documentCP happened more often (2672 times, 57%) than within-documentCP (2015 times, 43%). This finding concurs with previous work(only 35% of the CP events were within-document in [15]), makinga strong case for the importance of cross-document CP techniques.Units of text copied. Understanding the granularity and amountof text copied is important for designing AutoComPaste. Such information,however, has not been reported in literature. We empiricallycategorized the copy events into phrases (groups of 8 or lesswords), single sentences (groups of 8 or more words ending witha period), multiple sentences (at least one sentence without a newline),and paragraphs (one or more paragraphs, each ending with anewline).Surprisingly, while CP of phrases is common (39%), CP of oneor more sentences (33%) and paragraphs (28%) are also frequent.This finding suggests that a CP technique based on AC should supportdifferent granularity of text.Working context. Stolee et al. [15] found that word processorswere the most popular type of application while performing CP. Weextended the analysis a step further by analyzing the time intervalbetween CP events and typing in order to identify if CP occurs withtext editing. Empirically taking 30s as a threshold, we found that42% of all copy events were performed after a typing event, and54% of all paste events were followed by a typing event. Theseresults show that CP often occurs together with text editing.1617

+ Text Unit AdjustmentsWindow management is common and tediousCopy-paste often Interleaves typingCopy-paste different sizes of text is common

Show AutoComPaste Video17AutoComPaste Videohttp://www.youtube.com/watch?v=KoDT3UeAoRE18How does AutoComPaste Compare with Traditional Copy-Paste Techniques? 19

Ctrl-C, Ctrl-VMenu selectionDrag & dropX-WinChapuis and Roussel. Copy-and-paste between overlapping windows. CHI 07What are the conditions or factors?20In other words, what are the scenarios in which copy-paste is performed that will affect the performance of AutoComPaste and traditional copy-paste techniques? To find it out, we performed a bunch of pilot studies, and identified a number of factors. 2021

1) Knowledge of content Keyword(s) knownKeyword(s) unknown2) Knowledge of location Location knownLocation unknownI am going to use one of my own paper here so that I dont need to get permission from someone else.

2122

1) Knowledge of content Keyword(s) knownKeyword(s) unknown3) Visibility VisibleInvisible2) Knowledge of location Location knownLocation unknownI am going to use one of my own paper here so that I dont need to get permission from someone else.

2223

1) Knowledge of content Keyword(s) knownKeyword(s) unknown3) Visibility VisibleInvisible

4) Typing activity StandaloneInterleaving2) Knowledge of location Location knownLocation unknown23241) Knowledge of content Keyword(s) knownKeyword(s) unknown2) Knowledge of location Location knownLocation unknown3) Visibility VisibleInvisible4) Typing activity StandaloneInterleaving251) Knowledge of content Keyword(s) knownKeyword(s) unknown2) Knowledge of location Location knownLocation unknown3) Visibility VisibleInvisible4) Typing activity StandaloneInterleaving26

27

28

29

30

31

32

33

341) Knowledge of content Keyword(s) knownKeyword(s) unknown2) Knowledge of location Location knownLocation unknown3) Visibility VisibleInvisible4) Typing activity StandaloneInterleaving

35

S1: Content (known), Location (known), Visible (true), Typing before copy (false) 36

S1: Content (known), Location (known), Visible (true), Typing before copy (false) 37

S1: Content (known), Location (known), Visible (true), Typing before copy (false) 38

S1: Content (known), Location (known), Visible (true), Typing before copy (false) 39

S1: Content (known), Location (known), Visible (true), Typing before copy (false) 40

41

42

43

Controlled Experiment12 university participants X 2 techniques (XWin, ACP) X 2 content knowledge type (known, unknown) X 2 location knowledge type (known, unknown) X 2 visibility type (visible, invisible) X 2 pre-copy activity type (isolated, typing) X 6 trials of 3 different units of text (2 phrases + 2 sentences + 2 paragraphs)= 2304 trials total 44Results45

46

ACP has 29% performance benefitXWin has 29% performance benefitACP has 140% performance benefitXWin has 31% performance benefit

C(+) L(+)C(-) L(+)C(+) L(-)C(-) L(-)Show a video of performance advantages. 46Qualitative Study6 participants (3 female, 3 male; aged 22-25, mean 23.8)Realistic trip planning task plan a 5-day trip to Santa Barbara by gathering relevant information from 10 given webpagesasked to include at least one outdoor activity, one indoor activity, and one restaurant for each day of the tripCan use either AutoComPaste and other copy-paste techniques

47ResultsAutoComPaste is heavily used and highly rated by 5/6 participants

However, one rated AutoComPaste negatively He is a non-native English speaker participant48

ConclusionAutoComPaste nicely complements the traditional copy-paste techniquesAutoComPaste has advantage when the keyword/prefix is knownWhen keywords/prefix is known and location is unknown, AutoComPaste will have the most advantage XWin has advantage when the keyword/prefix is unknownPerformance of AutoComPaste is subject to typing and spelling skills

49This is demonstrated by the video. 49AcknowledgmentShi Xiaoming for programming the loggerGuia Gali and Symon Oliver for video editingStudy participants Members in the NUS-HCI labThis research is supported by National University of Singapore Academic Research Fund R-252-000-464-112

Q & A 51

Vignette (CHI 12)

You may want to check out these other projects from

SandCanvas (CHI 11)

MOGCLASS (CHI 11)

Magic Cards (CHI 09)

earPod (CHI 07)

Zone & Polygon Menu (CHI 06)

Elastic Hierarchy (InfoVis 05)

Simple Marking Menu (UIST 04)We are young lab, so I thought some advertisement doesnt hurt. If you like AutoComPaste, you may want to check out some of the other projects done by my lab and myself. Thank you and now I can take your questions. 51