`All I know is what I read in the newspaper’:

news/blog analysis with Lydia

Steven SkienaDept. of Computer ScienceStony Brook Universityhttp://www.cs.sunysb.edu/~skiena

Large-Scale Text Analysis Our Lydia text analysis system does a daily

analysis of over 1000+ online English and foreign-language newspapers, plus blogs, RSS feeds, historical archives, patents, and other news sources.

We currently track over 20,000,000 news entities in over one terabyte of text, providing spatial, temporal, relational and sentiment analysis.

We believe our data/analysis to be of great interest in social sciences and other domains.

Edison Chen and the Computer Technician

News Sentiment Time Series

International News Interest

International News Sentiment

Runner-up to Obama in Hong Kong’s 2008 “Man of the Year” poll!

www.textmap.com

The textmap.com website

http://www.textmap.com

TextMap Access

Outline of Talk

Introduction Lydia system architecture Spatial, temporal, and network analysis Sentiment analysis Applications Future directions

System Architecture

Spidering – text is retrieved from a given site on a daily basis using semi-custom spidering agents.

Normalization – clean text is extracted with semi-custom parsers and formatted for our pipeline

Text Markup – annotates parts of the source text for storage and analysis.

Back Office Operations – we aggregate entity frequency and relational data for a variety of statistical analyses.

Text Markup We apply natural language processing (NLP)

techniques to annotate interesting features of the document.

Full parsing techniques are too slow to keep up with our volume of text, so we employ shallow parsing instead.

We can markup approximately 2000 newspaper-days of text per CPU.

Analysis phases include…

Input

Dr. Judith Rodin, the former president of the University of Pennsylvania, will become president of the Rockefeller Foundation next year, the foundation announced yesterday in New York.

She will take over in March 2005, succeeding Gordon Conway, the foundation's first non-American president. Mr. Conway announced last year that he would retire at 66 in December and return to Britain, where his children and grandchildren live.

Sentence and Paragraph Identification

<p>Dr. Judith Rodin, the former president of the University of Pennsylvania, will become president of the Rockefeller Foundation next year, the foundation announced yesterday in New York.</p><p>She will take over in March 2005, succeeding Gordon Conway, the foundation's first non-American president.Mr. Conway announced last year that he would retire at 66 in December and return to Britain, where his children and grandchildren live.</p>

Part Of Speech Tagging<p>Dr./NNP Judith/NNP Rodin/NNP ,/, the/DT former/JJ president/NN of/IN the/DT University/NNP of/IN Pennsylvania/NNP ,/, will/MD become/VB president/NN of/IN the/DT Rockefeller/NNP Foundation/NN next/JJ year/NN ,/, the/DT foundation/NN announced/VBD yesterday/RB in/IN New/NNP York/NNP./.</p><p>She/PRP will/MD take/VB over/IN in/IN March/NNP 2005/CD ,/, succeeding/VBG Gordon/NNP Conway/NNP ,/, the/DT foundation/NN 's/POS first/JJ non-American/JJ president/NN ./.Mr./NNP Conway/NNP announced/VBD last/JJ year/NN that/IN he/PRP would/MD retire/VB at/IN 66/CD in/IN December/NNP and/CC return/NN to/TO Britain/NNP ,/, where/WRB his/PRP$ children/NNS and/CC grandchildren/NNS live/VBP ./.</p>

Proper Noun Extraction

<p><pn> Dr./NNP Judith/NNP Rodin/NNP </pn> ,/, the/DT former/JJ president/NN of/IN the/DT <pn> University/NNP </pn> of/IN <pn> Pennsylvania/NNP </pn> ,/, will/MD become/VB president/NN of/IN the/DT <pn> Rockefeller/NNP </pn> Foundation/NN next/JJ year/NN ,/, the/DT foundation/NN announced/VBD yesterday/RB in/IN <pn> New/NNP York/NNP </pn> ./.</p><p>She/PRP will/MD take/VB over/IN in/IN March/NNP 2005/CD ,/, succeeding/VBG <pn> Gordon/NNP Conway/NNP </pn> ,/, the/DT foundation/NN 's/POS first/JJ non-American/JJ president/NN ./.<pn> Mr./NNP Conway/NNP </pn> announced/VBD last/JJ year/NN that/IN he/PRP would/MD retire/VB at/IN 66/CD in/IN December/NNP and/CC return/NN to/TO <pn> Britain/NNP </pn> ,/, where/WRB his/PRP$ children/NNS and/CC grandchildren/NNS live/VBP ./.</p>

Actor Classification<p><pn category = "PERSON"> Dr./NNP Judith/NNP Rodin/NNP </pn> ,/, the/DT former/JJ president/NN of/IN the/DT <pn category = "UNKNOWN"> University/NNP </pn> of/IN <pn category = "STATE"> Pennsylvania/NNP </pn> ,/, will/MD become/VB president/NN of/IN the/DT <pn category = "UNKNOWN"> Rockefeller/NNP </pn> Foundation/NN next/JJ year/NN ,/, the/DT foundation/NN announced/VBD yesterday/RB in/IN <pn category = “CITY”> New/NNP York/NNP </pn> ./.</p><p>She/PRP will/MD take/VB over/IN in/IN <embedded_date> March/NNP 2005/CD </embedded_date> ,/, succeeding/VBG <pn category = "PERSON"> Gordon/NNP Conway/NNP </pn> ,/, the/DT foundation/NN 's/POS <num type = "ORDINAL"> first/JJ </num> non-American/JJ president/NN ./.<pn category = "PERSON"> Mr./NNP Conway/NNP </pn> announced/VBD last/JJ year/NN that/IN he/PRP would/MD retire/VB at/IN <num type = "CARDINAL"> 66/CD </num> in/IN <embedded_date> December/NNP </embedded_date> and/CC return/NN to/TO <pn category = "COUNTRY"> Britain/NNP </pn> ,/, where/WRB his/PRP$ children/NNS and/CC grandchildren/NNS live/VBP ./.</p>

Rewrite Rules<p><appellation> Dr. </appellation> <pn category = "PERSON"> Judith Rodin </pn> , the former president of the <pn category = "UNIVERSITY"> University of Pennsylvania </pn> , will become president of the <pn category = "UNKNOWN"> Rockefeller Foundation </pn> next year , the foundation announced yesterday in <pn category = “CITY”> New York </pn> .</p><p>She will take over in <embedded_date> March 2005 </embedded_date> , succeeding <pn category = "PERSON"> Gordon Conway </pn> , the foundation 's <num type = "ORDINAL"> first </num> non-American president .<appellation> Mr. </appellation> <pn category = "PERSON"> Conway </pn> announced last year that he would retire at <num type = "CARDINAL"> 66 </num> in <embedded_date> December </embedded_date> and return to <pn category = "COUNTRY"> Britain </pn> , where his children and grandchildren live .</p>

Alias Expansion<p><appellation> Dr. </appellation> <pn category = "PERSON"> Judith Rodin </pn> , the former president of the <pn category = "UNIVERSITY"> University of Pennsylvania </pn> , will become president of the <pn category = "UNKNOWN"> Rockefeller Foundation </pn> next year , the foundation announced yesterday in <pn category = “CITY”> New York </pn> .</p><p>She will take over in <embedded_date> March 2005 </embedded_date> , succeeding <pn category = "PERSON"> Gordon Conway </pn> , the foundation 's <num type = "ORDINAL"> first </num> non-American president .<appellation> Mr. </appellation> <pn category = "PERSON"> Gordon Conway </pn> announced last year that he would retire at <num type = "CARDINAL"> 66 </num> in <embedded_date> December </embedded_date> and return to <pn category = "COUNTRY"> Britain </pn> , where his children and grandchildren live.</p>

Geography Normalization<p><appellation> Dr. </appellation> <pn category = "PERSON"> Judith Rodin </pn> , the former president of the <pn category = "UNIVERSITY"> University of Pennsylvania </pn> , will become president of the <pn category = "UNKNOWN"> Rockefeller Foundation </pn> next year , the foundation announced yesterday in <pn category = “CITY, STATE, COUNTRY”> New York City, New York, USA </pn> .</p><p>She will take over in <embedded_date> March 2005 </embedded_date> , succeeding <pn category = "PERSON"> Gordon Conway </pn> , the foundation 's <num type = "ORDINAL"> first </num> non-American president .<appellation> Mr. </appellation> <pn category = "PERSON"> Gordon Conway </pn> announced last year that he would retire at <num type = "CARDINAL"> 66 </num> in <embedded_date> December </embedded_date> and return to <pn category = "COUNTRY"> Britain </pn> , where his children and grandchildren live.</p>

Back Office Operations

The most interesting analysis occurs after markup, using our Hadoop-based statistical analysis of all occurrences of interesting entities.

Each day’s worth of analysis yields about 10 million occurrences of about 1 million different entities, so efficiency matters...

Linkage of each occurrence to source and time facilitates a variety of interesting analysis.

M. Bautin, C. Ward, and S. Skiena, Lydia News/Blog Analysis, 2nd Hadoop Summit, Santa Clara CA, 2009.

Freedonia Architecture!Freedonia Architecture!

Eliminating the Database Bottleneck Separating NLP analysis from backend analysis

eliminates daily processing cycles. Distributed Map-Reduce processing with Hadoop

aggregates years of counts in under an hour on our cluster!

Such throughput facilitates large-scale evaluation to improve NLP/sentiment

Server architecture now oriented around time series vs. daily processing

Map-Reduce Processing

Distributed processing using the Hadoop implementation of Google’s MapReduce model

InputFiles

Map()

Map tasks

Map()

Map()

Reduce()

Reduce()

Reduce()

Reduce tasks

Outputs(key,

value)*

Hashing by key Recieves

(key, value*)

OutputFiles

Synonym Sets JFK, John Kennedy, John F. Kennedy, and John

Fitzgerald Kennedy all refer to the same person. We need a mechanism to link multiple entities that

have slightly different names but refer to the same thing.

We say that two actors belong in the same synonym set if:

There names are morphologically compatible. If the sets of entities that they are related to are

similar.➔Levon Lloyd, Andrew Mehler, and Steven Skiena. Identifying Co-referential Names Across Large Corpra. In Proc. Combinatorial Pattern Matching (CPM 2006)

Duplicate Article EliminationSupreme Court Justice David Souter suffered minor injuries when a group of young men assaulted him as he jogged on a city street, a court spokeswoman and Metropolitan Police said Saturday.

Supreme Court Justice David Souter suffered minor injuries when a group of young men assaulted him as he jogged on a city street, a court spokeswoman and Metropolitan Police said.

Hashing techniques can efficiently identify duplicate and near-duplicate articles appearing in different news sources.

Outline of Talk

Introduction Lydia system architecture Spatial, temporal, and network analysis Sentiment analysis Applications Future directions

HeatmapsWhere are people are talking about particular topics?

Newspapers have a sphere of influence based on: Power of the source – circulation, website popularity Population density of surrounding cities

The heat a given entity generates in a particular location is a function of the frequency it is mentioned in local sources

➔A. Mehler, Y. Bao, X. Li, Y. Wang, and S. Skiena. Spatial analysis of News Sources, IEEE Trans. Visualization (2006)

Who is talked about where?

Arnold Schwarzenegger Alabama

Donde Esta Mexico?

Which candidate got more press?

Historical News Analysis: Dual Presidencies in U.S. History

Juxtaposition Analysis

We compute the significance of the co-occurrences count between two entities in terms of their individual frequencies.

The number of raw co-occurrences biases towards relationships with popular entities.

Obama and his Friends Top Juxtapositions for Barack Obama

Juxtapositions between Barack Obama and John McCain

George Bush’s Social NetworkGeorge Bush’s Social Network

Entity Networks

Statistically-significant entity juxtapositions naturally define networks of news entities

Our experiments show that these networks are scale-free, i.e. observe power law distributions on both entity reference frequency and vertex degree.

These networks are useful for visualization, traversal, and more.

Discovering Entity Communities

Identifying natural communities of news entities facilitates interesting analysis.

E.g: Democrats, technologists, Italians, actors, baseball players, New Yorkers.

Curated rosters of such communities are usually unobtainable, incomplete, and/or ambiguous

A. Mehler and S. Skiena, Expanding Entity Communities from Representative Examples, ACM Trans. Knowledge and Data Discovery. 3 (2009)

Expanding Communities from Seeds Our approach: grow a large community from

a small seed set of known members. Selected representatives of any distinct

community are easy to identify (e.g. Wikipedia)

We incrementally expand the network to include the entity with the most significant in-group neighborhood.

Identifying Baseball Players

But when do we stop?

This incremental process eventually starts adding members from outside the community, destroying the result.

We partition our known members into seed and validation sets.

The gap between insertions of validation members grows substantially soon as we leave the community.

When is a baseball player not a baseball player?

Outline of Talk

Introduction Lydia system architecture Spatial, temporal, and network analysis Sentiment analysis Applications Future directions

Sentiment Analysis

Sentiment analysis lets us to measure how positively/negatively an entity is regarded, not just how much it is talked about.

Enron Sentiment Index, 1996-2005 (New York Times)

Blog Analysis with LydiaBlogs represent a different view of the world than newspapers.

Less objective Greater diversity of topics

We adapted Lydia to process Livejournal blogs, and compared blog content to that of newspapers.

➔Levon Lloyd, Prachi Kaulgud, and Steven Skiena. News vs. Blogs: Who Gets the Scoop?. In AAAI Spring Symposium: Computational Approaches to Analyzing Weblogs. 2006

Most Positive Actors in News and Blogs

News: Felicity Huffman, Fenando Alonso, Dan Rather, Warren Buffett, Joe Paterno, Ray Charles, Bill Frist, Ben Wallace, John Negroponte, George Clooney, Alicia Keys, Roy Moore, Jay Leno, Roger Federer

Blogs: Joe Paterno, Phil Mickelson, Tom Brokow, Sasha Cohen, Ted Stevens, Rafael Nadal, Felicity Huffman, Warren Buffett, Fernando Alonso, Chauncey Billups, Maria Sharapova, Earl Woods, Kasey Kahne, Tom Brady

Most Negative Actors in News and Blogs

News: Slobodan Milosevic, John Ashcroft, Zacarias Moussaoui, John Allen Muhammad, Lionel Tate, Charles Taylor, George Ryan, Al Sharpton, Peter Jennings, Saddam Hussein, Jose Padilla, Abdul Rahman, Adolf Hitler, Harriet Miers, King Gyanendra

Blogs: John Allen Muhammad, Sammy Sosa, George Ryan, Lionel Tate, Esteban Loaiza, Slobodan Milosevic, Charles Schumer, Scott Peterson, Zacarias Moussaoui, William Jefferson, King Gyanemdra, Ricky Williams, Ernie Fletcher, Edward Kennedy, John Gotti

How Do We Do it?

We use large-scale statistical analysis instead of careful NLP of individual reviews.

We expand small seed lists of +/- terms into large vocabularies using Wordnet and path-counting algorithms.

We correct for modifiers and negation. Statistical methods turn these counts into

indicies.➔N. Godbole, M. Srinivasaiah, and S. Skiena. Large-Scale Sentiment Analysis for News and Blogs. Int. Conf. Weblogs and Social Media, 2007

Good to Bad in Three Hops

Paths of WordNet synonyms can lead to contradictory results, requiring careful path selection.

What Does it Mean? Our scores corrolate very well with financial,

political, and sporting events.

Decreasing fear of AIDS in the scientific and popular press

Multilingual Sentiment Analysis Implementing language-specific sentiment analysis takes:

1. Language-specific NLP software (e.g. POS tagger, NER )

2. Language-specific linguistic resources (e.g. WordNet)

Machine translation is used in our English sentiment analysis

Foreign text TranslatorEnglish

sentimentanalyzer

M. Bautin, L. Venu-reddy, S. Skiena, International Sentiment analysis for news and blogs, IWCSM 2008

But what does it mean?

The key question here is assessment: Do meaningful sentiment signals survive the limitations of mechanical translation?

Motivations for multilingual sentiment analysis includeForeign market researchPublic opinion analysis

Approaches to Evaluation No gold standard exists for international sentiment

scoring (subjective, requires broad knowledge of international issues).

But we can correlate with real-world data such as stock indexes or baseball team results.

We can also assess whether we detect similar sentiment on contemporary news feels across languages and translators – uncorrelated results would mean no signal.

Summary of ResultsWe see consistent entity sentiment between

languages, measured through correlation. 43 out of 45 pairs correlated with p < 0.05.

Also consistent across different translators. Significant to p < 0.001.

Outline of Talk

Introduction Lydia system architecture Spatial, temporal, and network analysis Sentiment analysis Applications Future directions

Lydia Applications in the Social Sciences Ethnic group analysis. News sentiment and financial asset

pricing. The influence of media in politics Consumer attitudes and market research.

Name Ethnicity Classification

Family and given names convey considerable information about ethnicity.

We developed a HMM / decision tree classifier to assign people in the news to the most likely cultural/ethnic group.

Group trends become clear when applied to the 20 million people in our news data.

A. Ambekar, C. Ward, J. Mohammed, S. Male, and S. Skiena, Name-Ethnicity Classification from Open Sources, KDD 2009

Ethnicity HMM / Decision Tree

Accuracy by CEL Group

Aggregated Group AnalysisAggregated Group Analysis

Muslim CEL Group References, 1987-2004 (New York Times)

Though name ethnicity analysis and M/R group aggregation, we can monitor news coverage by nationality or other groups.

Identifying Differences in News Coverage Between Cultural/Ethnic Groups(with C. Ward and M. Bautin), News Analysis Workshop of IEEE/ACM Web Intelligence (WI 2009)

Lydia Analysis vs. Census Data

Frequency of coverage of entities with Hispanic names in the U.S. news, 2004-2008

Percentage of population self-reporting as Hispanic in the 2000 census. Courtesy of Wikipedia.

CEL U.S. Frequency Maps

Nordic

Italian

French

Hispanic Sentiment in the United States

We discovered a negative sentiment against Hispanics in regions where they are heavily represented, unlike other groups in our study.

Temporal References by CEL

Geographic Associations

Top geographic juxtapositions for two world leaders.

National classification results for heads of state and government.

Geographic Associations/CELs

(a) African (b) Hispanic (c) East Asian (d) Indian (e) Eastern

European (f) Muslim

CEL Group Associations

Tracking the number of co-references between different CEL groups provides insight into how strongly groups interact.

Movie Gross Modeling / Forecasting

Investors, movie studios, distributors, and exhibitors are all interested in forecasting movies’ grosses before their release.

Hollywood Stock Exchange http://www.hsx.com

Does coupling news analysis with traditional variables lead to better forecasting models?

W. Zhang and S. Skiena, Improving Movie Gross Prediction through News Analysis, IEEE/ACM Web Intelligence (WI 2009)

Frequency Count Example

Night at the Museum 1 Night at the Museum 2

Correlation AnalysisLogged News References vs. Logged Gross (or budget)

Correlation AnalysisLogged Sentiment Counts (Pre-release) vs. Logged Gross

Want to improve movie gross?Add extra violence!

Performance Comparison (Regression Model)

Performance Comparison (k-NN Models)

Stock Price Forecasting

Despite the Efficient Market Hypothesis, there is some evidence that prices do not optimally react to changes in news..

If so, sentiment/frequency data should be factored into pricing/trading models.

W. Zhang and S. Skiena, Trading Strategies to Exploit News Sentiment, in preparation.

News Frequencies vs. Stock Trading Volume

A Market-Neutral News-Based Trading Agent Method: Rank companies by their reported

sentiment each day, then go long (short) on equal amounts of positive (negative) sentiment stocks

Four key tunable parameters: n: The number of selected stocks s: The number of historical days used for sentiment

calculation. h: holding days Cl and Cu: The lower bound and upper bound of firms’

market capital.

Backtesting Results

Market Strategy Comparison

Our strategy shows positive returns with low volatility!

Almost very impressive: the timing resolution of our news data makes it unconfirmable that all news comes from before market opening…

Media Influence in Political Science Chicken and egg problem: does the public

influence media coverage or visa versa? Lydia provided high-resolution media

analysis for the 2008 National Annenberg Election Survey.

Elite opinion and the Iraq War (Huddy-Johnston-Lebo): Republicans reacted differently to war news than Democrats.

Swings in Sentiment: Obama vs. Clinton

General Sentiment

Technology licensed from Stony Brook SBIR and investor funding Eight employees and counting..

Understanding Popular Sentiment: Hummer vs. Gas Prices

Outline of Talk

Introduction Lydia system architecture Spatial, temporal, and network analysis Sentiment analysis Applications Future directions

Future Directions

Larger scale / improved data access, to support… Historical corpus / blog analysis Foreign-language news analysis Group-aggregated frequency analysis Improved sentiment analysis Meaningful collaboration with social scientists: e.g.

the sociology of cumulative advantage

Possible Areas for Collaboration

Visualization of news/blog analysis AI/Machine learning applied to Lydia data. Integrating/constructing entity ontologies. Identifying entity photoimages Cloud computing / MapReduce Domain-specific projects…

The Lydia Team

… and the Lydia Cluster

Sentiment Count Examples

Performance Comparison

Variations in Topic Coverage

Lydia classifies each article into news, business, sports, entertainment.

Thus we can study CEL group penetration into these different spheres.

Stock Return & Polarity Distribution

Cross-Language News Stream Analysis For each pair of languages, sentiment score

correlation is calculated for the common entities in these languages

Polarities are correlated much more strongly than frequencies or subjectivities

Bold correlations are significant with p < 0.05

Time Series News AnalysisTime Series News Analysis

• New York Yankees References, 1987-2004 (New York Times)

• Eliot Spitzer Sentiment Rank, Feb. 2008 - Mar. 2008 (US Newspapers)