Data Mining for Moderation of Social Data

Fernando G. Guerrero CEO SolidQ fguerrero@solidq.com

Introductions • Fernando G. Guerrero •Global CEO of SolidQ • fguerrero@solidq.com

•Microsoft Regional Director for Spain since 2004 • SQL Server MVP from year 2000 till 2007 •Usual suspect at many international conferences

SolidQ 2012… 10th anniversary •160 people in 23 countries:

• Argentina, Australia, Austria, Bulgaria, Canada, Chile, Costa Rica, Croatia, Denmark, France, Germany, India, Israel, Italy, Mexico, Saudi Arabia, Serbia, Slovakia, Slovenia, Spain, Sweden, UK, USA

•50 current or former RDs or MVPs •Authors of many books, articles, and whitepapers •Research Collaboration with:

• Universidad de Alicante • Universidad de les Illes Balears • Universidad de Santiago de Compostela • The European Union • The Spanish Ministry of Economy and Innovation

Agenda

• Social Data •Market Research • Sentiment Analysis, Text Mining •Moderation, Data Mining • SolidQ Research Lines in Social Data

Social data is everywhere

Social data is about everything

Social is there

• Is your organization promoting social about you?

Products Services Stories

Social is there, reputation

•What is social saying about you? • Product • Services • Decisions • Image

Market Research

•What is social requesting you? • Future Services • Product updates

•Can you ask questions to social?

• Is this service going to succeed • How can I fixed the current problem • Is society ready for this law

Sentiment Analysis, Text Mining

The movie was fabulous!

The movie stars Mr. X

The movie was horrible!

[ Factual ] [ Sentimental ] [ Sentimental ]

What is Data Mining?

• Inform actionable business decisions •Contrasts with “machine learning”

Media Case Study

•Millions of posts per year (different moderation scenarios) •About 25% are human moderated •About 10% of the moderated posts fail •No Business Intelligence applications for analysis

or reporting

Moderation, Data Mining

• Contextual Information • Time • Location • User

• At 10am comments are safer than at 2AM. • A user maybe safe talking about science bad

dangerous talking about sports. • If a thread is hot (dangerous), comment maybe hot. • Combining context pattern the systems assign risk to

posts without going into the text.

Solution – Logical Model

•Post Context (behavior analysis) • Patterns, data mining.

•Post Content (text analysis) • Profanity, low score sentences, text mining, mood or

tone (sentiment analysis)

Typically Available Data on Posts

•Historical and real time data for: • User (e.g. userid, email, nationalid) • Location (e.g. Life & Style Fashion) • Time (e.g. 12 March 2011 18:56) • Content (e.g. text, link, picture, video). • Moderation result

•Other attributes like geography, age, education could be used

Post context, Patterns, Data Mining •User behavior. • Time behavior. • Location behavior.

Building useful attributes • 1.- Thread ( % Fails in a certain thread) • 2.- User (% Fails per User) • 3.- Diff Hour Forum Created (TimeDatePosted-TimeForumCreated) • 4.- User Forum (% Fails in a certain forum) • 5.- Diff Last for User (TimeDatePosted - TimeLastFailUser) • 6.- Hour of the day • 7.- Diff hour UserJoined-Now (TimeDatePosted-TimeUserJoined) • 8.- User Thread (% Fails per User in a thread) • 9.- Diff Hour Thread Created (TimeDatePosted-TimeThreadCreated) • 10.- Day of Week • More than 100 attributes.

Hard Work •Periods. •Algorithms. •Algorithms' parameters. •Model refreshing. •Attribute analysis. •Outliers. •Overpopulating. •Behavior after this systems is in production.

Data Mining Algorithms

•Decision Trees/Linear Regression • Sequence Analysis •Neural Networks/Logistic Regression •Clustering • Text Mining (Words and Phrases)

Conclusion on Context

•Risk based on context of the post • Time • User’s history • Publish location

• Enables risk analysis for all type of content • Comments (in any language) • Links • Pictures • Videos

Logical Model: Post content

•Profanity Analysis • Text Mining

The first minister and his secretary found sleeping together last night. They got drunk at a nearby pub.

• Sentiment Analysis

Moderation, Data Mining System

Analysis and Reporting •Published through integrated web application

• Moderation statistics. • Users statistics. • News and Stories Statistics. • Peaks.

Conclusion: Benefits

•Moderating half of the total posts, the solution captures 90% of failing posts. The remaining 10% seem to be likely safe posts. •Using Intelligent Moderation, media companies

scan the whole universe of posts at a comparatively low cost. •At peak times, Intelligent Moderation works

perfect.

Football night in Europe

•On January 25th, 2012: • Liverpool defeated Manchester City in the Carling Cup • Barcelona defeated Real Madrid in Copa del Rey

•More than 100.000 comments arrived to the different BBC sites during 10 hours •All comments were filtered through our system •No problems observed during that time

SolidQ Team in this project

•Project Managers • Francisco Gonzalez, Javier Torrenteras, Alejandro

Leguizamo

•Developers • Itzik Ben-Gan, Enrique Puig, Ruben Pertusa, Carlos

Martinez , Fernando G. Guerrero

• Technical reviewers • Mark Tabladillo, Dejan Sarka

• Social Media Specialist. • Jose Quinto, Rocio Díaz

SolidQ Reseach

• Incomplete Grammar Analysis •Human interaction with IT systems

• Collaboration • Contextual analysis

• Sentiment Analysis • Market Research • Reputation

•Data Mining of context Social • Moderation • Market Research • Reputation

Invisible computing…

… Driven by Social Data

THANK YOU!

Fernando G. Guerrero Global CEO SolidQ fguerrero@solidq.com

Data Mining for Moderation of Social Data

Technology

September 4, 20151 Chapter 1. Introduction Motivation: Why data mining? What is data mining? Data Mining: On what kind of data? Data mining functionality

Data Mining Lecture 1: Introduction to Data Mining

Gufran Ahmad. Contents What is Data Mining? Data Mining / KDD process Different aspects of Data Mining Why Data Mining? Data Mining in Business Examples

UNIT - I Data Mining. UNIT - I Introduction : Fundamentals of data mining, Data Mining Functionalities, Classification of Data Mining systems, Major issues

MODERATION & MEDIATION October 23 rd, 2009 Download Data: - Peattie - Exam Anxiety

Introduction to Data Mining and Knowledge Discovery · PDF fileIntroduction to Data Mining and ... Introduction to Data Mining and Knowledge Discovery, ... Data mining and data warehousing

What is Data Mining? Data Mining Motivation Data Mining Applications Applications of Data Mining in CRM Data Mining Taxonomy Data Mining Techniques

Web Mining – Data Mining im Internet Mining – Data Mining im Internet Vorlesung SS 2010 ... Web Mining is Data Mining for Data on the World-Wide Web Text Mining: Application of

Data Mining BY JEMINI ISLAM. Data Mining Outline: What is data mining? Why use data mining? How does data mining work The process of data mining Tools

Introduction to Introduction to Data Mining Data Mining

1 Data Mining Chapter 26. 2 Chapter 1. Introduction Motivation: Why data mining? What is data mining? Data Mining: On what kind of data? Data mining functionality

NZIP Conference Physics Moderation. Outline Purpose of moderation Submitting material for moderation Physics moderation resources Using an assessment

October 18, 2015 Data Mining: Concepts and Techniques 1 DATA MINING Motivation: Why data mining? What is data mining? Data Mining: On what kind of data?

Secure Data Processing - uni-leipzig.de · PRIVACY-PRESERVING DATA MINING 12 Local Data Local Data Local Data Warehouse Data Mining Local Data Mining Local Data Mining Combiner Local

1 Chapter 1. Introduction Motivation: Why data mining? What is data mining? Data Mining: On what kind of data? Data mining functionality Major issues in

Applied Data Mining - Lagout Mining/Applied Data Mining [Xu... · The book reviews applied data mining from theoretical basis to ... data mining application arenas, ... research topics

Data mining and privacy preserving in data mining

Data Mining: What is Data Mining?

Web Mining – Data Mining im Internet Mining – Data Mining im Internet Vorlesung SS 2014 ... Web Mining is Data Mining for Data on the World-Wide Web Text Mining: Application of

Data Mining LECTURE # 01 Introduction to Data Mining