Got a Complaint? Keep Calm and Tweet It!...4 We publish the rst annotated dataset of public complaints and reports to the research community forbenchmarking and comparison 5 We evaluate

Got a Complaint? Keep Calm and Tweet It!

Nitish Mittal1, Swati Agarwal2, Ashish Sureka3

1Wadi.com, Dubai, UAE

2IIIT-Delhi, India

3ABB Corporate Research, Bangalore, India

International Conference on Advanced Data Mining and Applications (ADMA)Gold Coast, QLD, Australia

December 14, 2016

1 / 41

Introduction

Anonymity

Wide Reachability

Connectivity

Low Barriers

Popularity • Complaints and Grievances • Reports- Frauds, Bribe, Corruption • Bringing Government’s attention to

Public Issues

Government Bodies & Concerned Authorities

Low Publication Barriers

• Bad Road Conditions

Figure 1: Reaching-out to Public and Addressing Citizen-Centric Problems

2 / 41

Introduction

Figure 2: Concrete Examples of Citizen Complaints Reported to Government’s OfficialTwitter Handlers- Tweets address various public issues such as traffic violation,inconvenience in train coach.

3 / 41

Steps Taken by Indian Government

1 TwitterSevatelecom and postal sectors consumers and stake-holders complaints mentioning@manojsinhabjp- Official Twitter handler of Telecom Minster Mr. Manoj Sinha.#DOTSeva or #BSNLSeva or #MTNLSeva or #PostalSeva

2 CybercellComplaints against women online trolls and right wing abuseInitiative taken by @Manekagandhibjp- Union Minister for Women and ChildDevelopment, Govt. of India.

3 MociSevaa part of TwitterSevaconsumer needs assistance from the Ministry of Commerce & Industry

4 / 41

Response?

5 / 41

Response?

6 / 41

Response?

7 / 41

Response?

8 / 41

Related Work

Article ObjectiveHeverin et al. [8] examine the use of Twitter by city police departments in large U.S. cities

that have active Twitter accounts.

Anderson et al. [3] study on Twitter adoption across American municipal police departmentsserving populations over 100K

Meijer et al. [12] empirical analysis of Twitter usage by the Dutch police officers for ex-ternal communication

Edwards et al. [5] investigating 4 cases of webcare of Dutch public organizations by ad-dressing the client feedback and related sentiments

Vanessa et al. [7] analyzing the behavioral similarities and differences of 3-1-1 phone ser-vice (formal) and Twitter (informal) channels for reporting issues in thecommunity

9 / 41

Technical Challenges (Application, Platform and Analytics)

PlatformMassive Size and High VelocitySpam and Fake Accounts

ApplicationHighly Imbalance DataData Annotation and Ground TruthAmbiguous and sarcastic postsManipulation, Fabrication and Adversarial Behavior

AnalyticsMultilingualism (script and language)Noisy Content (spell and grammar errors, abbreviations, slangs)Use of Multimedia formats

10 / 41

Research Contributions

1 We build a text analysis based ensemble classifier for identifying complaints andgrievances reports from non-complaint tweets.

2 We apply core natural language processing techniques and address the challengeof presence of noisy content in the dataset.

3 We publish the first enhanced and enriched database of citizens’ complaintstweets. We make our data publicly available for benchmarking, extension andcomparison1 [1].

1https://data.mendeley.com/datasets/w2cp7h53s5/111 / 41

https://data.mendeley.com/datasets/w2cp7h53s5/1

Dataset Collection

Method: Twitter REST APIDuration: four weeks (11 April 2016 to 8 May 2016)Keyword: @username

1 @RailMinIndia (Railway Ministry of India)

2 @dtpTraffic (Delhi Traffic Police)

3 @DelhiPolice (Delhi Police)

4 @IncomeTaxIndia (Income Tax Department, Government of India)

12 / 41

Dataset Characterization- Multi-media Content

RailMinIndia dtptraffic DelhiPolice IncomeTaxIndia0

5k

10k

15k

20k

25k

30k

35k

Num

ber

of

Tw

eets Original

Filtered

Sampled

36182

31123

1500 15241307

100010001295

1720 383327

200

(a) Number of Tweets in Experimental Dataset

RailMinIndia dtptraffic DelhiPolice IncomeTaxIndia0

200

400

600

800

1000

1200

1400

Num

ber

of

Tw

eets

Users

Hashtags

Images

URLs

1365

147262

145

662

119333

72

681

275

202323

15848

2150

(b) Contextual Metadata Statistics

Figure 3: Experimental Dataset Statistic- Illustrating the statistics of number of tweetscollected, filtered and sampled for each account. Further, showing the variation innumber of sampled tweets consisting of various entities (URL, hashtag, image,@usermention)

13 / 41

Data Pre-processing

Topic Identification Word Count Features (WCF) Named Entities Features (NEF)

Religion

Unknown

Concepts

Multi-Label

Taxonomy Posts tagged with all religions mentioned in the textual content

LIWC PTopic

Feature 1 Feature 2

..

..

.. Feature N

Posts

User

NER

NER

PNEF

UNEF

Experimental Dataset

Post Metadata Post

Metadata

Post and Blogger Metadata

Multi-Class Classifier

Class 1 Class 2

..

..

.. Class N

Conflict Identifier

Labeled Data

Raw Tweets WCF, PNEF

Feature Space

Yes No

UNEF PTopic Feature Space

Data Enhancement and Enrichment

Hashtag Expansion

Sentence Segmentation

Spell Error Correction

Acronyms and Slang Treatment

Enriched Tweets

Figure 4: High Level Framework Design of Data Enhancement and Enrichment- Theproposed framework addresses the challenge of noisy data in the tweets by expandingjoint hashtags, normalizing text, correcting spelling errors and translating slang andabbreviations in a raw tweet.

14 / 41

Hashtag Expansion

1 Common Separator (’ ’ and ’-’)

#no-strong-action-by-police = no strong action by police#black money = black money

2 Uppercase Letters

#MarchForDemocracy = March For Democracy#CharchaOnRWH = Charcha On RWH#FANTomorrow = FAN Tomorrow

3 Alphanumeric String

#TheriJoins100crClub = Theri Joins 100 cr Club

4 Porter Stemming

#seriousissue = serious issue#havesomesenseofchecking = have some sense of checking#swachhbharat #swaranshatabdi #modisarkar

15 / 41

Spell Error Correction

Given Sentence S= ”pleas answr my query asaap”,set of 3-grams= [n1: ’pleas answr my’, n2: ’answr my query’, n3: ’my query asaap’]

16 / 41

Acronyms and Slang Treatment

1 Domain Specific Slangs

Rly- Railway, NH- National Highway, tkt- ticket, acc- accident, wtng- waiting, exp-express

2 Numerics

2- to, two, too?4- for, four?

3 Others

B4- before, plz- please, idk- i don’t know, c- see

17 / 41

Appreciation Posts

18 / 41

Promotion and Advertisement

19 / 41

News Update or Information Sharing

20 / 41

Features Identification

1 Frequent N-grams (1 x n)

2 Events and Substances (1)

3 Location (1)

4 Media Presence (1)

21 / 41

Frequent N-grams

1 Defines the topic of the post

2 Overcomes the limitations of Keyword Based Flagging

3 Grouping of similar terms using lexical knowledgebase

Account N-Grams Grouped Triplets@DelhiPolice bribe, abuse, harrasment, FIR, phone, action,

report, complaint

@dtpTraffic bribe, challan, violation, abuse, harassment,jam, commotion, congestion, accident

@RailMinIndia train number, train name, coach, pnr number,bribe, corruption, report, complaint, action

@IncomeTaxIndia pan number, ack, FIR, TIN, complaint, re-port, investigation, refund

22 / 41

Events and Substances

1 complaints that do not affect the reporters directly

2 reported to bring the attention of public agencies towards common concerns

3 extracted using IBM Watson ’Concept and Relationship Extraction API’

Example 1

traffic violation, playing loud music in vehicle, alcohol consumption in trains

23 / 41

Location Information

Example 2

An accident happened on Ring Road towards Mathura Road (along the Ashramflyover) ORWe are waiting since 3 hours on platform of Delhi safdarganj due to the delay ofjammu-tavi train

Example 3

M.B.Road, Gandhi Nagar, AIIMS Metro Station, PVR Saket

24 / 41

Media Presence

25 / 41

Complaint and Grievance Tweet Classification

1 Split our data (classified as unknown after AISP classification) into 1:3 ratio

2 Train our model on linguistic and contextual features3 Investigate the efficacy of feature vectors by running our classifier for different

types of SVM kernels [2]

LinearPolynomialRBF

4 Boosting of Base-line Approach

26 / 41

Empirical Analysis and Experimental Results

@dtpTraffic @RailMinIndia @IncomeTaxIndia0

0.2

0.4

0.6

0.8

1


Pre

cisi

on

Linear

Polynomial

RBF

Cascaded Ensemble

Parallel Ensemble

0.7

6

0.7

5

0.4

7

0.5

6

0.7

6

0.4

3

0.4

2

0.2

8

0.3

3

0.4

3

0.6

2

0.6

1

0.3

6

0.4

3

0.8

3

27 / 41

Infographic Visualization of Complaints and Grievances Reports

28 / 41

Conclusions

1 Emergence of Social Media platforms (Twitter) for reporting citizens’ complaintsand grievances

2 We address the challenge of noisy and free-form text by enhancing the usergenerated data

3 Proposing various linguistic and contextual features for filtering non-complainttweets

4 We publish the first annotated dataset of public complaints and reports to theresearch community for benchmarking and comparison

5 We evaluate the performance of our features by varying the kernel parameters ofSVM classifier

6 We boost the accuracy of base-line approach by using cascaded and parallelensemble learning

29 / 41

Future Work

1 Identification of ambiguous reports2 Improvising the accuracy of linguistic features and addressing the limitations of

present study1 Hashtag Expansion2 Query Reports3 Domain Knowledge4 Sarcastic Posts

3 Developing a front-end visualization tool and creating a dashboard of publiccomplaints

30 / 41

Thank You!

Email: [email protected]

Web: www.swati-agarwal.in

31 / 41

References I

[1] Swati Agarwal, Nitish Mittal, and Ashish Sureka.Enhanced dataset of citizen centric complaints and grievances on twitter, mendeley data, v1 http://dx.doi.org/10.17632/w2cp7h53s5.1, 2016.

[2] S. Amari and S. Wu.Improving support vector machine classifiers by modifying kernel functions.Neural Networks, 12(6):783 – 789, 1999.

[3] Megan Anderson, Kieran Lewis, and Ozgur Dedehayir.Diffusion of innovation in the public sector: Twitter adoption by municipal police departments in the us.In Portland International Conference on Management of Engineering and Technology, 2015.

[4] Wilas Chamlertwat, Pattarasinee Bhattarakosol, Tippakorn Rungkasiri, and Choochart Haruechaiyasak.Discovering consumer insight from twitter via sentiment analysis.J. UCS, 18(8):973–992, 2012.

[5] Arthur Edwards and Dennis Kool.Webcare in Public Services: Deliver Better with Less?, pages 151–166.Springer International Publishing, Cham, 2015.

[6] Stephen Few.Information dashboard design.2006.

[7] Vanessa Frias-Martinez, Abson Sae-Tang, and Enrique Frias-Martinez.To call, or to tweet? understanding 3-1-1 citizen complaint behaviors.In ASE BigData/SocialCom/CyberSecurity Conference, 2014.

[8] Thomas Heverin and Lisl Zach.Twitter for city police department information sharing.Proceedings of the American Society for Information Science and Technology, 2010.

[9] Bernard J Jansen, Mimi Zhang, Kate Sobel, and Abdur Chowdury.Twitter power: Tweets as electronic word of mouth.Journal of the American society for information science and technology, 60(11):2169–2188, 2009.

32 / 41

References II

[10] Gohar Feroz Khan, Bobby Swar, and Sang Kon Lee.Social media risks and benefits a public sector perspective.Social Science Computer Review, 32(5):606–627, 2014.

[11] Euripidis Loukis, Yannis Charalabidis, and Aggeliki Androutsopoulou.Evaluating a Passive Social Media Citizensourcing Innovation, pages 305–320.Springer International Publishing.

[12] Albert Jacob Meijer and René Torenvlied.Social media and the new organization of government communications an empirical analysis of twitter usage by the dutch police.The American Review of Public Administration, page 0275074014551381, 2014.

[13] Bharath Sriram, Dave Fuhry, Engin Demir, Hakan Ferhatosmanoglu, and Murat Demirbas.Short text classification in twitter to improve information filtering.In Proceedings of the 33rd international ACM SIGIR conference on Research and development in information retrieval, pages 841–842. ACM,2010.

33 / 41

Dataset Characterization- Popular User Mentiions

Table 1: Statistics of Tweets Containing Direct Mentions to Related OfficialGovernment Twitter Handles

Account Tweets Popular Mentions@DelhiPolice 1000 @CPDelhi (131), @ArvindKejriwal (97), @HMOIndia

(44), @PMOIndia (34), @narendramodi (26)@RailMinIndia 1500 @suresh prabhu (720)@IncomeTaxIndia 1000 @arunjaitely@dtptraffic 1000 @ArvindKejriwal (139)

34 / 41

Dataset Characterization- Topic Modeling

Hashtag Count Topic Hashtag Count Topic

@DelhiPolice @dtpTraffic#OddEven 14 Vehicle Rule #OddEven 53 Vehicle Rule#IPSAKnowledgeSeries 13 IPS Discussion #OddEvenDobara 44 Vehicle Rule#sexualharassment 6 Harassment #kotlamubarakpur 3 Car on Fire

@IncomeTaxIndia @RailMinIndia#SovereignUnnathi 2 Construction Project #RailDrishti 15 Initiative#Theri 2 Raid #Latur 10 Relief Operation#Aadhaar 1 Unique ID #RailwayZoneForVizag 8 Metro

35 / 41

Dataset Characterization- Topic Modeling

Table 2: Concrete Examples of Frequently Occurring 7 and 8 Character-gram Strings inthe Experimental Dataset of Each Public Service Account

Account Hashtags@DelhiPolice traffic, missing, abusing, arrested, detained, criminal, communal

@dtpTraffic flyover, oddeven, parking, pillion, crossing, redlight, hospital

@IncomeTaxIndia website, efiling, pending, invoice, property, interest, marriage, passport

@RailMinIndia toilets, sleeper, medical, delayed, cleaning, security, drinking, stoppage

36 / 41

Data Pre-processing

Table 3: Examples of Original and Enriched Complaint Tweets Before and AfterPerforming the Text Pre-processing and Enhancement of the Content

Before AfterHashtagExpansion

@IncomeTaxIndia unearths Rs 52.5cr #black-money from Amritsar Rice miller via @time-sofindia

unearths Rs 52.5cr black money from AmritsarRice miller via.

@ArvindKejriwal#OddEvenDobara being brokenin broad daylight 2 men inside @abpnewstv @dtp-Traffic

Odd Even Dobara being broken in broad daylight2 men inside.

Spell Errormaterial snt by railways n 21/3. Current sta-tus:railways saying we haven’t gt material.

Material sent by railways n 21/3. Current sta-tus:railways saying we haven’t got material.

my frnd geting call frm 8757969668 claiming tob from Naptol asking to dpsit 12500 so that theywill deliever Safari car.

My friend getting call from 8757969668 claimingto be from naaptol asking to deposit 12500 sothat they will deliver safari car.

Slang Ex-pansion

Sir y is it so dat the vendor in S-9 of shramjeeviexp. Is taking more charge on a bottle of amulkool?

Sir why is it so that the vendor in S-9 of shram-jeevi express Is taking more charge on a bottle ofamul cool?

Pls tel n c JI dat failure of laws on cops led todis in US learn from it.

Please tel and see JI that failure of laws on copsled to this in US learn from it.

37 / 41

Appreciation, Information Sharing and Promotion (AISP) TweetsClassifier

Table 4: Concrete Examples of Non-C&G Tweets- Presence of appreciation, query,promotion and information sharing content in a tweet are strong indicators of a tweetfor certainly not being a complaint report.

Category TweetAppreciation Sir thanks for releasing all Income tax return forms and utilities in month of April as promised

by you Kudos to. Good Governance.

Query Will Odd Even apply if i take u turn from under rajokri flyover 2 go 2 ambience mall fromIFFCO. Gurgaon.

Promotion Income Declaration Scheme: Government assures complete confidentiality:http://goo.gl/jVU5qK @FinMinIndia @IncomeTaxIndia

Information Sharing @RailMinIndia Unravelling the Central Railway’s hidden history.http://www.thehindu.com/todays-pape... @sureshpprabhu

38 / 41

Complaints and Grievance Tweets Classification

CDK N-Grams Location Event Media

Topic Identification Word Count Features (WCF) Named Entities Features (NEF)

Religion

Unknown

Concepts

Multi-Label

Taxonomy Posts tagged with all religions mentioned in the textual content

LIWC PTopic

Feature 1 Feature 2

..

..

.. Feature N

Posts

User

NER

NER

PNEF

UNEF


Post Metadata Post

Metadata

Post and Blogger Metadata

Multi-Class Classifier

Class 1 Class 2

..

..

.. Class N

Conflict Identifier

Labeled Data

WCF, PNEF

Feature Space

Yes No

UNEF PTopic Feature Space

Testing Data

Labeled Data

NER

CDK

Concepts

Substance

Fi+1 .. .. Fn-1 Fn

Fi+1 .. .. Fn-1 Fn

Feature1 .. .. Featurei-1 Featurei

Tone

Core NLP Sampled

Data (ED1)

Feature1 .. .. Featurei-1 Featurei

Tone

Core NLP

Information Sharing

Promotion Appreciation

AISP Tweet Classification AISP

Unknown (ED2)

ED2

Testing Data

Training Data

Location

CDK

Events, Substance

N-grams

Fi+1 .. .. Fn-1 Fn

Fi+1 .. .. Fn-1 Fn Media Presence

Linear

Polynomial

RBF

Linguistic

Contextual

Features Identification

One-Class SVM Classification

C&G (ED3)

Unknown (ED4)

POS Patterns

Query Identifier

Query Tweets

Unknown ED4

Topic Modeling

Complaint Type Identifier

ED3

Label1 .. .. Labeli-1 Labeli Multi-Labeling

1

2 3

4 5

Figure 5: A General Research Framework for Complaints and Grievances TweetClassification

39 / 41


Table 5: Confusion matrix Results for C&G Tweets Classifiers (SVM with 3 differentkernel parameters)- Tables illustrate the number of true positives, true negatives andfalse alarms generated by these classifiers.

SVM Kernel TP TN FP FN Precision Recall F-Score

@dtpTrafficLinear 184 294 58 178 0.76 0.51 0.61

Polynomial 170 296 56 192 0.75 0.47 0.58RBF 182 146 206 180 0.47 0.50 0.48

@DelhiPoliceLinear 19 382 226 24 0.08 0.44 0.14

Polynomial 18 387 221 25 0.08 0.42 0.13RBF 27 221 387 16 0.07 0.63 0.13

@IncomeTaxIndiaLinear 31 52 19 21 0.62 0.60 0.61

Polynomial 22 57 14 30 0.61 0.42 0.50RBF 25 27 44 27 0.36 0.48 0.41

@RailMinIndiaLinear 188 471 252 123 0.43 0.60 0.50

Polynomial 139 534 189 172 0.42 0.45 0.43RBF 149 340 383 162 0.28 0.48 0.35

40 / 41


Table 6: Confusion matrix Results for Ensemble SVM Classifiers (Combining 3 SVMclassifiers in Cascaded and Parallel Fashion)

Ensemble TP TN FP FN Precision Recall F-Score

@dtpTrafficSerial 314 104 248 48 0.56 0.87 0.68

Parallel 41 338 14 321 0.76 0.11 0.20

@DelhiPoliceSerial 41 63 545 2 0.07 0.95 0.13

Parallel 6 536 72 37 0.08 0.14 0.10

@IncomeTaxIndiaSerial 46 11 60 6 0.43 0.88 0.58

Parallel 5 70 1 47 0.83 0.10 0.17

@RailMinIndiaSerial 253 206 517 58 0.33 0.81 0.47

Parallel 41 659 64 270 0.39 0.13 0.20

41 / 41

IntroductionTechnical ChallengesResearch ContributionsExperimental SetupData Enhancement and EnrichmentClassificationExperimental ResultsConclusions

Documents

Got a Complaint? Keep Calm and Tweet It!...4 We publish the rst annotated dataset of public complaints and reports to the research community forbenchmarking and comparison 5 We evaluate