8/13/2019 God Does Not Play Dice with Social Sentiments

1/2

God Does Not Play Dice with Social SentimentsW ALID S. S ABA

I previously wrote an article [1] expressing my opinion on the many claims that have been made inrecent years by a flood of so-called Sentiment Analysis software products. My main argument wasthat sentiment analysis is even a much more difficult and challenging problem than that of buildingcomputer programs that can understand ordinary spoken language. Since we do not, as of yet, have asolution to the latter, we certainly do not have anything that can decipher sentiment from ordinaryspoken language, and what these system do is just slightly better than flipping a coin. I outlined in thatarticle that the state-of-the-art in natural language processing (NLP) is still not at the point where itcan deal with metonymy, metaphor, sarcasm, irony, non-literal meaning, etc. The article received somegood comments and some that essentially questioned the frequency of such phenomena in ordinaryspoken language. In a follow-up article [2] I replied to these concerns showing some data that

illustrates that metonymy, metaphor, sarcasm, irony, non-literal meaning, etc. are not rare in ordinarylanguage use, but are the norm. Given that these phenomena constitute more than 50% of languageuse, and given that the best accuracy any statistical approach has claimed is around 80%, then theaccuracy of any sentiment analysis software is less than that of flipping a coin, as has also beenreported by a formal study [3].

I can understand the excitementand eagerness of some that want toachieve this monumentalchallenge, but things have gone outof hand. We now have many withlittle or no experience or evenknowledge of the many challengesof building computational modelsfor language analysis makingclaims that are beyond anycredibility. In the long run, this is very damaging to the whole enterprise of NLP. When the truth comes out about the vacuous claims ofthese systems, it will negatively taint the entire NLP community and this will hinder progress in realNLP work that is based on solid scientific and logical foundations.

I do not have the space nor the opportunity here to make a scientific argument on the vacuous andfalse claims of such software systems (this will appear shortly elsewhere!). I will however briefly

present here an existential proof of my claim. I have conducted a thorough experiment using severalof the leading sentiment analysis software systems, using text written by both, those that vehementlyadvocate and defend the validity of such products, as well as those that question the entire enterprise.

Paradoxically, when presented with articles written by those that are very excited about automaticsentiment analysis, sentiment analysis software was assigned a negative sentiment, even when theinput was content written by the vendors of such products! What is more bizarre, is that sentimentanalysis software received a positive sentiment when presented with text that is very critical of suchproducts (e.g., this article!) Negatives and positives were almost assigned at a whim, and many

8/13/2019 God Does Not Play Dice with Social Sentiments

2/2

software products did not agree on sentiment. The results were really what I expected: it was almost aflip of a coin.

I am not going to report here on the software products I used in my experiment, nor on the articlesand authors that very strongly advocate such software products. I could if I need to, but I invite thedefenders of such software products to try their articles on the software of their choice!

Little Knowledge is Dangerous

When I was a graduate student working on my PhD in AI/NLP, I was fortunate to have been seated in the reception dinner of ACL-95 (which was at held at MIT) on the same table that one ofthe most prominent researchers in AI/NLP was seated. He was atBell Labs, back then, and he is now at the [one of?] largestsoftware companies on the globe. I quietly and discretely asked:

I have looked at this years conference proceedings, and I do notsee any real language understanding work no new semanticformalisms, theories, or models. All I see are Markov models, theinfamous Bayesian formula, and large data/corpus analysis experiments with some tabulated results with good (of course!) precision and recall numbers at the end. Whats going on? The answer to mylong question was honest and clear: We all gave up! We cannot figure out what the semantic rules ofordinary spoken language are, not to mention our complete ignorance of the pragmatic, epistemic andcognitive aspects of ordinary spoken language. So, we now crunch data and hope to discover somepatterns!

I can understand the frustration of researchers that constantly needed to produce publishable results. Ican even understand the over eagerness of some hackers who clearly underestimate the real challengesin computational language understanding, but we are now at point where almost every day there issomeone that claims they have some software that can quantify opinions, attitudes, feelings, andsentiments expressed in ordinary spoken language. This is damaging for all of us who really work inNLP.

1. Walid Saba (2012), Henry Kissinger vs. Sentiment Analysis, SEOJournal.2. Walid Saba (2012), Henry Kissingers Sentiments are not an Exception, SEOJournal.3. Brian Tarran (2010), Automated Sentiment Analysis Gives Poor Showing in Accuracy Test, research-live.com.