The Dynamics of Topical Conversations in Microblogging

Victoria Lai and William RandSocialCom 2013September 11, 2013

Keyword and trend identification Background noise Importance of time, subject, geography?

2

Twitter’s trending topics Real-time trend identification◦ Mathioudakis and Koudas (2010)◦ Cataldi, Di Caro, and Schifanella (2010)

Other trend detection methods◦ Benhardus and Kalita (2013)

Trends in a geographic area◦ Wilkinson and Thelwall (2012)

3

Daily Twitter REST API queries, 47 weeks Topics◦ 11 keywords/phrases chosen for time, subject, and

geographic variety (baseball, economy, global warming, love, London riots)◦ 5 control (I, the, and, a, of)

Geography◦ Global◦ 9 cities (Boston, D.C.,

London, Sydney)

4

Term frequency

Inverse document frequency

TF-IDF

Keyword lists as target/control sets vary Time, subject, geography Tf-idf and rank correlation

𝑓𝑓 𝑇𝑇𝑡𝑡∗, 𝑠𝑠∗, 𝑔𝑔∗ 𝐶𝐶𝑡𝑡, 𝑠𝑠, 𝑔𝑔 )

6

…

tf phrase 1 #phrase 2 #phrase 3 #C0, ∀sc, ∀g

C1, ∀sc, ∀g

C2, ∀sc, ∀g

C3, ∀sc, ∀g

idf

phrase 1 #phrase 2 #phrase 3 #

phrase 1 #phrase 2 #phrase 3 #

…

T0, s*, ∀g

* SC = {I, the, and, a, of }

Control set variation by time◦ Single week◦ Cumulative week

Control set variation by geography Target set variation by time

8

Topic matters, time doesn’t.t

9

Top keywords are very consistent, particularly

for subjective topics.

10

The correlation decays as we add tweets from another week back in time.

11

12

For trends relative to the current background noise, we only need a few weeks of control data.

13

Trends are independent of local background noise and more dependent on global background noise.

14

Some topics are more similar to background conversations than others.

15

Subjective topics exhibit less vocabulary change over time.

16

The results of this analysis provide suggestions for how to build a trend monitoring tool.

These recommendations help define and understand: Frequency of collection Short-term vs. long-term collections Local collections vs. global collections Keyword selection for best results

17

Subject variation Other methods of comparison Blogging data

18

19