Upload
nathen-boyce
View
214
Download
0
Tags:
Embed Size (px)
Citation preview
1
Daniel Avrahami – Doctoral Symposium – UIST’06
Who, What, and When:Supporting Interpersonal Communication over Instant Messaging
Daniel AvrahamiCarnegie Mellon University
www.cs.cmu.edu/~nx6
2
Daniel Avrahami – Doctoral Symposium – UIST’06
Illustration
John is making final changes to a presentation for a client visit. His team member Anne, working at a different site, sends him an instant message asking for some urgent information.
3
Daniel Avrahami – Doctoral Symposium – UIST’06
Illustration
John is making final changes to a presentation for a client visit. His team member Anne, working at a different site, sends him an instant message asking for some urgent information.
Since John is pressed for time, he decides to ignore all incoming messages until after he’s done, leaving Anne unable to finish her task.
4
Daniel Avrahami – Doctoral Symposium – UIST’06
Illustration (cont)
Consider now if we were able to: Accurately predict, based on his activity, that John
was not likely to respond to Anne’s message for some time
Predict, based on past communication patterns, that Anne and John are co-workers
Such models could be used, for example, to increase the salience of the alert, indicating to John that Anne’s message may deserve his immediate attention
5
Daniel Avrahami – Doctoral Symposium – UIST’06
Research goals
The two main goals of my research work are to provide a better understanding of factors affecting IM interaction in its context, and to use this understanding for the creation of predictive statistical models and tools that support IM communication.
In order to achieve these goals my research will use three complementary steps:
6
Daniel Avrahami – Doctoral Symposium – UIST’06
Research goals
Create accurate models that predict responsiveness to incoming IM, and investigate the factors affecting responsiveness (when)
Investigate the effect of interpersonal relationships on IM interaction, and create statistical models that use this knowledge to predict relationships (who)
Use basic properties of human dialogue to provide support for balancing of responsiveness and performance (what)
7
Daniel Avrahami – Doctoral Symposium – UIST’06
Background
Instant Messaging, or IM, is one of the most popular communication mediums today 12 billion instant messages are sent each day. Nearly 1
billion messages are exchanged by 28 million business users [IDC Market Analysis’05]
Useful in many ways: from quick questions and clarifications, coordination and scheduling, to discussions of complex work [Bradner’99; Nardi’00; Handel’02; Herbsleb’02; Isaacs’02]
IM has a number of shortcomings (Asynchrony + Limited aware + Low cost for sending)
8
Daniel Avrahami – Doctoral Symposium – UIST’06
When:Predicting responsiveness to IM
[Presented at CHI’06]
9
Daniel Avrahami – Doctoral Symposium – UIST’06
Background
Wanted to answer the following question:
If an instant message were to arrive right now, would the user respond to it? In how long?
10
Daniel Avrahami – Doctoral Symposium – UIST’06
Data Collection
11
Daniel Avrahami – Doctoral Symposium – UIST’06
Data collection
Created a plugin for Trillian Pro (written in C) Non-intrusive collection of:
IM events desktop events
12
Daniel Avrahami – Doctoral Symposium – UIST’06
Participants
16 participants to date Nearly 5200 hours recorded Over 90,000 messages Over 400 buddies 4 participants provided full text
On average, participants exchanged a message every: 3.4 minutes (researchers avg=8.1)
13
Daniel Avrahami – Doctoral Symposium – UIST’06
Responsiveness
0 50 100 150 200 250 300 350 400 450 500
Message Number
Day
Hour
10 min5 min2 min1 min30 sec
92%
14
Daniel Avrahami – Doctoral Symposium – UIST’06
Defining “Session Initiation Attempts”
0 50 100 150 200 250 300 350 400 450 500
Message Number
Day
Hour
10 min5 min2 min1 min30 sec
session
used two subsets: 5 minutes (similar to Isaacs’02) and 10 minutes
92%
15
Daniel Avrahami – Doctoral Symposium – UIST’06
What are we predicting?
Generate a set of features for every message: IM state and desktop state
“Seconds Until Response” computed, for every incoming message from a
buddy, by noting the time it took until a message was sent to the same buddy
Examined five responsiveness thresholds 30 seconds, 1, 2, 5, and 10 minutes
16
Daniel Avrahami – Doctoral Symposium – UIST’06
Modeling method
Weka ML toolkit Features selected using a wrapper-based
selection technique AdaBoosting on Decision-Tree models 10-fold cross-validation
10 trials: train on 90%, test on 10% Next I report combined accuracy
17
Daniel Avrahami – Doctoral Symposium – UIST’06
79.883.8
87.089.4 90.1
0
10
20
30
40
50
60
70
80
90
100
30sec 1min 2min 5min 10min
Predict response within
% A
ccu
rate
Results (full feature-set models)
All significantly better than the prior probability (p<.001)
(Graph shows 5-minute
subset only)
18
Daniel Avrahami – Doctoral Symposium – UIST’06
Results (buddy-independent models)
Previous models used information about the buddy (e.g., time since messaging that buddy)
Can predict different responsiveness for different buddies But what if you wanted just one level of
responsiveness?
Built models that did not use any buddy-related features
19
Daniel Avrahami – Doctoral Symposium – UIST’06
79.882.5
87.0 89.4 89.3
Use
r C
entr
ic
Use
r C
entr
ic
Use
r C
entr
ic
Use
r C
entr
ic
Use
r C
entr
ic
0
10
20
30
40
50
60
70
80
90
100
30sec 1min 2min 5min 10min
Predict response within
% A
ccu
rate
Results (buddy-independent models)
all significantly better than the prior probability (p<.001) BUT not sig. diff. from previous set
20
Daniel Avrahami – Doctoral Symposium – UIST’06
Some practical considerations
Preserving plausible deniability
Making predictions about the receiver, visible to the receiver
Multiple concurrent levels of responsiveness
21
Daniel Avrahami – Doctoral Symposium – UIST’06
Who:Relationships and IM Communication
[To be presented at CSCW’06]
22
Daniel Avrahami – Doctoral Symposium – UIST’06
Relationships and IM communication
People use IM for both work and social communication
Prior research shows that relationship type has significant effect on fact-to-face and other voice communication (Duck’91)
Wanted to investigate the effect of relationship on basic communication patterns
23
Daniel Avrahami – Doctoral Symposium – UIST’06
Co-worker (Senior) Co-worker (Peer) Co-worker (Junior) Co-worker (Other) Friend Family Spouse Significant Other Acquaintance Friend & Co-worker Self Bot [Unknown/Unused]
Buddy Coder
24
Daniel Avrahami – Doctoral Symposium – UIST’06
Session-level measures
# Time Message Text
1 17:42:45 B: Hey [Participant’s name]
2 17:42:56 B: what time does your group get in the AM?
3 17:42:57 P: hey
4 17:43:01 P: usually around 10
5 17:43:25 B: ok
6 17:43:38 B: i want to start circulating the card in the AM
7 17:43:58 P: ok, good idea
8 17:44:02 P: that's for coordinating this
9 17:44:13 B: no problem
10 17:44:27 P: thanks :-)
11 17:44:35 P: sorry bout the typo
12 17:44:38 B: is ok
Variable Value
Group Student
Relationship Work
Duration 1.88 minutes
Message Count 12
Turn Count 7
Character Count 232
Messages per Minute 6.4
Messages per Turn 1.71
Characters per Message 19.3
Seconds Until First Reply 1 seconds
Minimum Gap (between turns) 1 seconds
Maximum Gap (between turns) 24 seconds
Average Gap (between turns) 12.2 seconds
Time of Day 5:44 pm
25
Daniel Avrahami – Doctoral Symposium – UIST’06
The effect of relationships
Used a repeated-measures ANOVA Relationship Category (Work, Mix, Social) and
Group (Researchers, Interns, Students) were repeated
Participants and BuddyID modeled as random effects
Participants nested in Group BuddyID nested first in Participants, then in
Group
N=3297 sessions
26
Daniel Avrahami – Doctoral Symposium – UIST’06
Results
27
Daniel Avrahami – Doctoral Symposium – UIST’06
Summary of Results
Sessions with Social contacts were longer and with more messages BUT at a significantly slower pace Maybe giving less attention to these sessions?
Sessions with Work contacts were at a faster pace with longer messages Grounding? Complex concepts?
28
Daniel Avrahami – Doctoral Symposium – UIST’06
Results: Session length
Significant effect on Session Duration (p<.001)
Social significantly longer sessions than both Mix and Work (Work and Mix n.s.)
Similar effects forNumber of TurnsNumber of MessagesNumber of Characters(Duration correlated at >.85)
0
1
2
3
4
5
6
7
8
w ork mix social
Relationship
Du
rati
on
(m
inu
tes)
29
Daniel Avrahami – Doctoral Symposium – UIST’06
Results: Messaging rate
Significant effect on Messaging Rate (p<.01) Social significantly slower than Mix (p=.003) Social marginally slower than Work (p=.078)
Maximum-Gap (p<.05)Social longer than Work(p=.013)
0
1
2
3
4
5
6
7
w ork mix social
Relationship
Mes
sag
es-p
er-M
inu
te
30
Daniel Avrahami – Doctoral Symposium – UIST’06
Results: Length of messages
Significant effect on Message Length (Characters-per-Message) (p<.001) Work significantly longer than both Social (p<.001) and Mix
(p=.002)
0
5
10
15
20
25
30
35
40
45
w ork mix social
Relationship
Ch
arac
ters
-per
-Mes
sag
e
31
Daniel Avrahami – Doctoral Symposium – UIST’06
Predicting relationships
32
Daniel Avrahami – Doctoral Symposium – UIST’06
Predicting relationships
How can it be used? Augmenting IM systems
Indicators of unavailability Differential alerts
Shared with other mediums E.g. Email
Provide organizational overview
33
Daniel Avrahami – Doctoral Symposium – UIST’06
Models performance
2-step Logistic regression model with 16-fold cross validation
Results from pairs with 2 sessions or more (78% of the data) Significantly better than the prior probability
Classified as
Work Social
Work40.9%(83)
5.9%(12)
Social14.8%(30)
38.4%(78)
Accuracy: 79.3%
Classified as
Work Mix Social
Work25.3%(74)
5.1%(15)
2.0%(6)
Mix8.2%(24)
14.7%(43)
7.8%(23)
Social9.6%(28)
17.1%(50)
10.2%(30)
Overall Accuracy: 50.2%Work vs. Rest: 75.1%Social vs. Rest: 63.5%
34
Daniel Avrahami – Doctoral Symposium – UIST’06
What:Using content to balance
responsiveness and performance
[Presented at CSCW’04]
35
Daniel Avrahami – Doctoral Symposium – UIST’06
Responsiveness / Performance Tradeoff
Users often multitask when using instant messaging [Nardi’00, Isaacs’02, Voida’02]
Users often have to choose between Staying on task and being responsive to IM
Current solutions typically force users to choose one or the other: Update ‘away’ messages Turn off IM client
36
Daniel Avrahami – Doctoral Symposium – UIST’06
Quick response - “do you have the figures?”
Leisurely response - “check out www.cnn.com”
Politely deferred - “ru busy?”
No response - “going to meeting. ttyl”
Expectations for responsiveness
37
Daniel Avrahami – Doctoral Symposium – UIST’06
The approach: QnA
Users ignore, to the best of their ability, the alerts of incoming messages Transitioning (internally) to being unavailable
By observing the content of messages, QnA automatically highlights incoming messages that may deserve their attention In particular, potential questions and answers
38
Daniel Avrahami – Doctoral Symposium – UIST’06
Why questions and answers?
A question and an answer form an ‘Adjacency pair’ (Schegloff & Sacks’73)
From “Arenas of Language Use”“Given a first pair part, a second pair part is conditionally relevant, that is, relevant and expectable, as the next utterance.
Once A has asked the question, it is relevant and expectable for B to answer in the next turn.”
(Clark 1992, p. 157)
39
Daniel Avrahami – Doctoral Symposium – UIST’06
How does it work?
QnA listens to incoming and outgoing messages when an outgoing messages is sent
if it is a question remember that expecting a response
when an incoming messages arrives if it is a question and/or we are expecting an answer
wait x seconds to see if user attends to the message if did not attend then show QnA notification
40
Daniel Avrahami – Doctoral Symposium – UIST’06
is_a_question?
Match to list of questions that can be ‘politely deferred’ (are|r) (you|u) there busy?
Go through list of rules and look for match (?|/) at end of sentence what (is|are|r|were|does|do|did|should|can) did(|n’t|nt) (i|u|you|he|she|they|we) (are|r) (you|u) huh
41
Daniel Avrahami – Doctoral Symposium – UIST’06
QnA summary
QnA: A tool that allows users to stay on task, but still seem responsive to buddies who expect it
Allows users to transition between work modes Sits quietly in the background when the user
attends to messages Only notifies when the user ignores messages
Download from www.cs.cmu.edu/~nx6
42
Daniel Avrahami – Doctoral Symposium – UIST’06
Conclusions & Future Work
43
Daniel Avrahami – Doctoral Symposium – UIST’06
Conclusions
I have presented work on analysis and generation of predictive modeling in support of interpersonal communication over IM:
Work on predictions of responsiveness to IM communication (specifically to session initiation attempts)
Work on analysis and predictions of interpersonal relationships and their effect on communication
Work on the use of basic properties of human dialogue to allow users to balance responsiveness and performance
44
Daniel Avrahami – Doctoral Symposium – UIST’06
Current and Planned Work
Understanding Responsiveness Investigate in detail the contribution of specific features
Investigate distribution of responsiveness over time
Content Analysis Determining the Communication Goals Content-based transcript segmentation
45
Daniel Avrahami – Doctoral Symposium – UIST’06
this work was funded in part by NSF Grants IIS-0121560, IIS-0325351, and by DARPA Contract No. NBCHD030010
thank you
for more info visit: www.cs.cmu.edu/~nx6
or email: [email protected]
46
Daniel Avrahami – Doctoral Symposium – UIST’06
Process diagram
47
Daniel Avrahami – Doctoral Symposium – UIST’06
Process diagram
48
Daniel Avrahami – Doctoral Symposium – UIST’06
Process diagram
49
Daniel Avrahami – Doctoral Symposium – UIST’06
Process diagram
50
Daniel Avrahami – Doctoral Symposium – UIST’06
Related work
Interruptions and disruptions [Gillie’89 , Cutrell’01 , Hudson’02 , Dabbish’04]
Interruptibility and cost of interruption [Horvitz’99 , Horvitz’03, Hudson’03 , Begole’04, Horvitz’04,
Fogarty’05, Iqbal’06]
Models of presence [Horvitz’02, Begole’03]
Responsiveness to Email [Horvitz’02, Tyler’03]
51
Daniel Avrahami – Doctoral Symposium – UIST’06
Participants
16 participants to date
Researchers: 6 full-time employees at an industrial research lab (mean age=40.33)
Interns: 2 summer interns at the industrial research lab (mean age=34.5)
Students: 8 Masters students (mean age=24.5)
52
Daniel Avrahami – Doctoral Symposium – UIST’06
How can such models help?
sender receiver
intercept alert mask enhance
awareness
message
53
Daniel Avrahami – Doctoral Symposium – UIST’06
sender
How can such models help?
message
receiver
intercept alert mask enhance
54
Daniel Avrahami – Doctoral Symposium – UIST’06
sender
How can such models help?
message
receiver
intercept alert mask enhance
55
Daniel Avrahami – Doctoral Symposium – UIST’06
sender
How can such models help?
message
receiver
intercept alert mask enhance
56
Daniel Avrahami – Doctoral Symposium – UIST’06
sender
How can such models help?
awareness
receiver intercept alert mask enhance
shhhh
57
Daniel Avrahami – Doctoral Symposium – UIST’06
sender
How can such models help?
awareness
receiver
intercept alert mask enhance (carefully)
not now
58
Daniel Avrahami – Doctoral Symposium – UIST’06
Data collection (cont.)
Privacy of data Masking messages
for example, the message:“This is my secret number: 1234 :-)” was recorded as “AAAA AA AA AAAAAA AAAAAA: DDDD :-)”.
Temporary masking
Alerting buddies
Hashing buddy-names
59
Daniel Avrahami – Doctoral Symposium – UIST’06
Who:Relationships and IM Communication
[To be presented at CSCW’06]
60
Daniel Avrahami – Doctoral Symposium – UIST’06
Relationships and IM communication
People use IM for both work and social communication
Availability might depend on relationship
Wanted to investigate the effect of relationship on basic communication patterns
61
Daniel Avrahami – Doctoral Symposium – UIST’06
Background
Relationship type has significant effects on communication, including the quality, purpose and perceived value [Duck’91]
Cues, such as tempo, pauses, speech rates and the frequency of turns, affect the way in which conversation partners perceive each other [Feldstein’94]
Frequency affects communication [FTF:Whittaker’94, IM:Isaacs’02]
62
Daniel Avrahami – Doctoral Symposium – UIST’06
Relationships distribution
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
Researchers Interns Students
Work
Mix
Social
Other
63
Daniel Avrahami – Doctoral Symposium – UIST’06
Results
Relationship Category
Work Mix Social Analysis of Variance
Variables Mean StdErr Mean StdErr Mean StdErr F d.f. p
Duration (in minutes) 4.0 0.6 5.2 0.6 6.6 0.5 8.04 2/331 <.001
Message count 13.8 3.0 19.8 3.1 25.9 2.8 6.11 2/398 <.01
Turn count 8.8 1.7 12.2 1.7 15.3 1.6 5.96 2/374 <.01
Character count 459.5 122.7 673.6 123.6 844.6 115.2 4.71 2/340 <.01
Messages-per-Minute 6.0 0.5 6.2 0.4 4.6 0.4 4.75 2/99 <.05
Messages-per-Turn § 1.5 0.05 1.5 0.05 1.6 0.05 2.32 2/312
Characters-per-Message 37.9 2.5 31.5 2.5 30.1 2.4 7.85 2/229 <.001
Seconds Until First Reply 36.9 3.0 35.0 3.1 36.0 2.7 0.11 2/151
Minimum Gap (between turns) 12.0 1.8 12.4 1.9 12.1 1.6 0.02 2/111
Maximum Gap (between turns) 68.7 3.8 77.0 3.9 81.8 3.4 3.25 2/173 <.05
Average Gap (between turns) 28.8 2.2 28.3 2.3 29.2 2.0 0.10 2/181
Time of Day § 14.6 0.4 14.6 0.4 14.7 0.4 0.04 2/253
64
Daniel Avrahami – Doctoral Symposium – UIST’06
0 (Work)
0 (Work)
1 (Social)
1 (Social)
0 (Work)b1i
b1
b1
b2
b2
0 (Work)
0 (Work)
1 (Social)
0 (Work)
1 (Social)
1 (Social)
0 (Work)
0 (Work)
0 (Work)
< v1,v2,...,vn >
< v1,v2,...,vn >
< v1,v2,...,vn >
< v1,v2,...,vn >
< v1,v2,...,vn >i
i
i
i
SN Buddy
b1i 3 .333 Yes
2 .5 Nob2i
SN nBuddy
ActualSessionVariables
Predicting relationships
Cross-validation with 16 models (omitting one participant each time)
Nominal Logistic Regression
65
Daniel Avrahami – Doctoral Symposium – UIST’06
What:Using content to balance
responsiveness and performance
[Presented at CSCW’04]
66
Daniel Avrahami – Doctoral Symposium – UIST’06
Issues
Determining that a message contains a question or an answer can be difficult interleaved conversations many short messages that comprise a single turn loose grammar and spelling
Gives buddies a way to increase the salience of their messages. what if they abuse it?
67
Daniel Avrahami – Doctoral Symposium – UIST’06
Future work
Collect feedback from users A few users who have used QnA for over 2 years
now But would like more users
Please download QnA from my homepage
Improve question identification
Implement ‘ignore list’
68
Daniel Avrahami – Doctoral Symposium – UIST’06
Conclusions & Future Work
69
Daniel Avrahami – Doctoral Symposium – UIST’06
Contributions
This work’s contribution to the HCI field will span both theoretical and applied aspects. From a theoretical point of view, this work will
provide insights into the factors that influence interpersonal communication patterns and responsiveness.
At the applied level, this work will provide predictive statistical models that can be used in many applications.
Finally, this work promotes the creation of tools that use knowledge and predictive models generated from naturally occurring interaction.