Sentiment Analysis - Columbia University · • Social Media Features *Apoorv Agarwal, Fadi Biadsy,...

SentimentAnalysis

Announcements•  HW1isduetodayat11:59pm•  HW2willcomeoutMonday•  IfyouuselatedaysforHW1,MUSTSTATEITINYOURSUBMISSION.•  Reading:Today:C4.2NLP• Monday:C8.1-8.3SpeechandLanguage,

8.1NLP•  RecommendreadingchaptersinYoavGoldbergonneuralnetsandJurafskyandMarVnaswell.•  BotharemoreintuiVve

Today• SenVmentanalysistasks:definiVon• SenVmentresources• TradiVonalsupervisedapproach• Neuralnetapproach

Whatissentiment?• ExpressionofposiVveornegaVveopinions• ..Towardsatopic,person,event,enVty• ..Towardsanaspect(e.g.,service,foodorambienceinarestaurant)

Whysentimentanalysis?• SenVmentiscommoninonlinepla[orms• Peoplewriteabouttheirpersonalviewpoints

• UsefultounderstandwhatpeoplethinkaboutpoliVcalissues,poliVcalcandidates,importanteventsoftheday• UsefulforgeneraVngsummariesofreviews:restaurants,products,movies

Thesentimentanalysistask(s)• SubjecVvevsobjecVve• Polarity:PosiVve,negaVveorneutral• DowehavesenVmenttowardsatarget?OraspectbasedsenVment?• What/whoisthesenVmentsource?

SubjectivevsObjective• �Atseveraldifferentlayers,it’safascina3ngtale.[“Who’sSpyingonOurComputers”,GeorgeMelloanWallStJournal.(Bookreview)• BellIndustriesIncincreaseditsquarterlyto10centsfrom7centsashare.

ExamplesfromWeibeetal2004

Positive/Negative/Neutral•  FromUseNet:

• NegaVve:Ihadinmindyourfacts,Buddy,nothers.•  PosiVve:Nicetouch.“Alleges”whateverfactspostedarenotinyourpersonaofwhatis“real”• Neutral:Marchappearstobeanes3matewhileearlieradmissioncannotbeen3relyruledout,"accordingtoChen,alsoTaiwan'schiefWTOnego3ator

ExamplesfromWeibeetal2004andRosenthal2014

SubjectivePhrases• TheforeignministrysaidThursdaythatitwas“surprised,toputitmildly”bytheU.S.StateDepartment’scri0cismofRussia’shumanrightsrecordandobjectedinparVculartothe“odious”secVononChechnya.[MoscowTimes,03/08/2002]• Subjec3vityanalysisiden3fiestextthatrevealsanauthor’sthoughts,beliefsorotherprivatestates.

ExamplesfromWeibeetal2004

SubjectivePhrasesandSources• TheforeignministrysaidThursdaythatitwas“surprised,toputitmildly”bytheU.S.StateDepartment’scri0cismofRussia’shumanrightsrecordandobjectedinparVculartothe“odious”secVononChechnya.[MoscowTimes,03/08/2002]• Whowassurprised?• WhowascriVcal?

ExamplesfromWeibeetal2004

SentimenttowardsTarget•  IpreUymuchenjoyedthewholemovie.Target=wholemovie,senVment=posiVve.•  Bulgariaiscri3cizedbytheEUbecauseofslowreformsinthejudiciarybranch,thenewspapernotes.Target=Bulgaria,senVment=negaVve•  Stanishevwaselectedprimeministerin2005.Sincethen,hehasbeenaprominentsupporterofhiscountry’saccessiontotheEU.Target=country’saccesstotheEU,senVment=posiVve

ExamplesfromBreck&Cardie

Datasets(Sem-evaldatasetsalsoused)

2000sentencesineachcorpus

12

Corpus AverageWordCount

AverageCharacterCount

Subjec6vePhrases

Objec6vePhrases

VocabularySize

CharacterLengthRestric6ons

LiveJournal 14.67 66.47 3035(39%) 4747(61%) 4747 30-120

MPQA 31.64 176.68 3325(41%) 4754(59%) 7614 none

Twimer 25.22 118.55 2091(36%) 3640(64%) 8385 0-140

Wikipedia 15.57 77.20 2643(37%) 4496(63%) 4342 30-120

MPQA:extensivelyannotateddatasetbyStoyanav,CardieandWeibe2004.15opinionorientedqusVons,15factorientedquesVons.Alongwithtextspansfrom252arVcles.

RosenthalandMcKeown2013)

ExampleSentences

LiveJournal iwillhavetosVcktomycanonfilmslrunVlinafewyearsicanaffordtoupgradeagain:)

MPQA ThesaleinfuriatedBeijingwhichregardsTaiwananintegralpartofitsterritoryawaiVngreunificaVon,byforceifnecessary.

Twimer RT@tashjade:That’sreallysad,CharlieRT“UnVltonightIneverrealisedhowfuckedupIwas”-CharlieSheen#sheenroast

Wikipedia PerhapsifreportedcriVcallybyawesternsourcebutcertainlynotbyanIsraelisource.

13

SubjecVve ObjecVve

SentimentLexicons• GeneralInquirer• SenVWordNet• DicVonaryofAffect(DAL)

DictionaryofAffectinLanguage• DicVonaryof8742wordsbuilttomeasuretheemoVonalmeaningoftexts• Eachwordisgiventhreescores(scaleof1to3)•  pleasantness-alsocalledevaluaVon(ee)•  acVveness(aa)•  andimagery(ii)

C.M.Whissel.1989.Thedic6onaryofaffectinlanguage.InR.PlutchikandH.Kellerman,editors,EmoVon:theoryresearchandexperience,volume4,London.Acad.Press.

15

Emoticons• 1000emoVconsweregatheredfromseverallistsavailableontheinternet• Wekeptthe192emoVconsthatappearedatleastonceandmappedeachemoVcontoasingleworddefiniVon

16

Methods• Pre-processingsteps•  EmoVconkeysandcontracVonexpansion• Chunkerandtagger*• LexicalFeatures*• SyntacVcFeatures*• SocialMediaFeatures

*ApoorvAgarwal,FadiBiadsy,andKathleenR.McKeown.2009.Contextualphrase-levelpolarityanalysisusinglexicalaffectscoringandsyntac6cn-grams.InProceedingsofEACL’09

17

PreprocessingLiveJournal [i]/NPsub[willhavetos6ck]/VPobj[to]/PPobj[mycanonfilmslr]/NPobj[unVl]/

PPobj[in]/PPobj[afewyears]/NPsub[i]/NPsub[canaffordtoupgrade]/VPobj[again:)]/NPsub

MPQA [Thesale]/NPsub[infuriated]/VPobj[Beijing]/NPobj[which]/NPsub[regards]/VPsub[Taiwan]/NPobj[anintegralpart]/NPsub[of]/PPobj[itsterritoryawaiVngreunificaVon,]/NPobj[by]/PPobj[force]/NPsub[if]/obj[necessary.]/sub

Twimer [RT@tashjade:]/NPobj [That]/Npobj[is]/VPsub[really]/sub[sad,]/sub[CharlieRT]/NPobj[”]/NPobj[UnVl]/PPobj[tonight]/NPsub[I]/NPsub[never]/sub[realised]/VPsub[how]/sub[fucked]/VPsub[up]/PPobj[I]/NPsub[was]/VPsub[”]/obj[-CharlieSheen#sheenroast]/NPobj

Wikipedia [Perhaps]/sub[if]/obj[reported]/VPsub[cri6cally]/sub[by]/PPobj[awesternsourcebut]/NPsub[certainlynot]/sub[by]/PPobj[anIsraelisource.]/NPsub

18Xuan-HieuPhan,CRFChunker:CRFEnglishPhraseChunkerhmp://crfchunker.sourceforge.net/,2006

LexicalFeatures• POSTags*• N-grams*• Performedchi-squarefeatureselecVononthen-grams

*ApoorvAgarwal,FadiBiadsy,andKathleenR.McKeown.2009.Contextualphrase-levelpolarityanalysisusinglexicalaffectscoringandsyntac6cn-grams.InProceedingsofEACL’09

19

SyntacticFeatures• Usethemarkedupchunkstoextractthefollowing:*• n-grams:1-3words• POS:NP,VP,PP,JJ,other• PosiVon:target,right,lev•  SubjecVvity:subjecVve,objecVve• Minandmaxpleasantness

*ApoorvAgarwal,FadiBiadsy,andKathleenR.McKeown.2009.Contextualphrase-levelpolarityanalysisusinglexicalaffectscoringandsyntac6cn-grams.InProceedingsofEACL’09

20

SocialMediaFeatures

21

SingleCorpusClassiLication

BalancedUnbalanced

•  LogisVcRegressioni

•  10runsof10-foldcross-validaVon

•  StaVsVcalsignificanceusingthet-testwithp=.001

22

Experiment LiveJournal MPQA Twimer Wikipedia

n-gramsize 100 2000 none none

majority 58% 59% 64% 63%

JustDAL 76.5% 75.7% 83.6% 80.4%

DicVonaries+SM 77.1% 76.1% 84% 81.4%

Wordnet 76.7% 75.6% 84% 80.7%

Wordnet+SM 77.1% 76.1% 84.2% 81.4%

DicVonaries 76.6% 75.7% 83.9% 80.7%

SM 77% 76.1% 83.7% 81.2%

Experiment LiveJournal MPQA Twimer Wikipedia

n-gramsize 100 200 none none

majority 50% 50% 50% 50%

JustDAL 74.7% 75.7% 81.9% 79.3%

DicVonaries+SM 76.7% 76.2% 82.6% 80.2%

Wordnet 75.1% 75.8% 82.4% 79.1%

Wordnet+SM 76.6% 75.3% 82.6% 80.3%

DicVonaries 75.3% 75.8% 82.4% 79.1%

SM 76.2% 76.3% 82.2% 80.4%

SocialMediaErrorAnalysis• Wikipedia• PunctuaVonwasusefulasafeaturefordeterminingthataphraseisobjecVveifitisasmallphrase.However,severalsubjecVvephraseswereincorrectlyclassifiedbecauseofthis 23

0%

20%

40%

60%

80%

100%

Wikipedia

subjecVveobjecVve

SocialMediaErrorAnalysis• Twimer•  EllipseshelpindicatethatasentenceisobjecVve.Theaccuracyimprovedfrom82%to92%forsentenceswiththisfeature•  AllothersocialmediafeatureswereincorrectlyclassifiedasobjecVve/subjecVvedependingonthesocialmediapreference. 24

0%

20%

40%

60%

80%

100%

TwiZer

subjecVveobjecVve

SocialMediaErrorAnalysis• LiveJournal•  OutofVocabularywordsandpunctuaVonwerethemostusefulsocialmediafeatures.•  InalldatasetsthepunctuaVonfeaturecausedcloseto50/50exchangebutthefeaturewasbestinLiveJournal. 25

0%

20%

40%

60%

80%

100%

Livejournal

subjecVveobjecVve

NeuralNetworkApproachestoSentiment•  TakeastandardRNN•  Takealabeleddataset(e.g.,IMDBsenVmentdataset)•  IniValizewithpre-trainedwordembeddings(wordtovecorglove)• UsesigmoidtopredictbinarysenVmentlabels:posiVvevsnegaVve.

Languageismadeupofsequences• Sofarwehaveseenembeddingsforwords•  (andmethodsforcombiningthroughvectorconcatenaVonandarithmeVc)

• Buthowcanweaccountforsequences?• Wordsassequencesoflemers•  Sentencesassequencesofwords• Documentsassequencesofsentences

RecurrentNeuralNetworks• Representarbitrarilysizedsequencesinfixed-sizevector• GoodatcapturingstaVsVcalregulariVesinsequences(ordermamers)• IncludesimpleRNNs,Longshort-termmemory(LSTMs),GatedRecurrentUnit(GRUs)

[Thangetal.2013] [Bowmanetal.2014]

Learningwordmeaning Logicalentailmentfromtheirmorphsusingcompositional

semanticsviaRNNs

MachineTranslation(Sequences)

• Sequence-to-sequence•  Sutskeveretal.2014

RNNAbstraction• RNNisafuncVonthattakesanarbitrarylengthsequenceasinputandreturnsasingledoutdimensionalvectorasoutput•  Input:x1:n=x1x2…xn(xiεRd-in)•  Output:ynεRd-out

OOutputvectoryusedforfurtherpredicVon

RNNCharacteristics• CancondiVonontheenVresequencewithoutresorVngtotheMarkovassumpVon• Cangetverygoodlanguagemodelsaswellasgoodperformanceonmanyothertasks

RNNsaredeLinedrecursively• BymeansofafuncVonRtakingasinputastatevectorhi-1andaninputvectorxi• Returnsanewstatevectorhi• ThestatevectorcanbemappedtoanoutputvectoryiusingasimpledeterminisVcfuncVon• AndfedthroughsovmaxforclassificaVon.

wx

wh

RecurrentNeuralNetworks

x1

h0

h1

ℎ↓𝑡 =𝜎( 𝑊↓ℎ ℎ↓𝑡−1 + 𝑊↓𝑥 𝑥↓𝑡 )

σ

SlidefromRadev

wx

wh

RNN

x1

h0

h1

ℎ↓𝑡 =𝜎(𝑊↓ℎ ℎ↓𝑡−1 + 𝑊↓𝑥 𝑥↓𝑡 )𝑦↓𝑡 =𝑠𝑜𝑓𝑡𝑚𝑎𝑥( 𝑊↓𝑦 ℎ↓𝑡 )

σ

y1so[maxwy

SlidefromRadev

RNN

wx

wh

x1

h0

h1

σwx

wh

x2

h2

σwx

wh

x3

h3

σ

y3so[max

The cat sat

wy

SlidefromRadev

UpdatingParametersofanRNN

wx

wh

x1

h0

h1

σwx

wh

x2

h2

σwx

wh

x3

h3

σ

y3so[max

The cat sat

Cost

wy

BackpropagaVonthroughVme

SlidefromRadev

Example• Foreachsentenceinthetrainingcorpus,classify,comparetogoldstandardandcomputeloss,backpropagate.• Recallthatwemayusemini-batchessothatwe’renotback-propagaVngforeachexample

•  Ihadinmindyourfacts,Buddy,nothers.

RNN–Ihadinmindyourfacts,buddy,nothers.

wx

wh

x1

h0

h1

σwx

wh

x2

h2

σwx

wh

x3

h3

σ

y3sigmoid

I had in

wx

wh

x3

h3

σ

mind …

Inthisoverview,wreferstotheweightsButtherearedifferentkindsofweightsLet’sbemorespecific

RNN–Ihadinmindyourfacts,buddy,nothers.

wx

U

x1

h0

h1

wx

U

x2

h2

σwx

U

x3

h3

σ

y3sigmoid

I had in

wx

U

x3

h3

σ

mind …

σ

RNN–Ihadinmindyourfacts,buddy,nothers.

wx

U

x1

h0

h1

wx

U

x2

h2

σwx

U

x3

h3

σ

y3sigmoid

I had in

wx

U

x3

h3

σ

mind …

σ

RNN–Ihadinmindyourfacts,buddy,nothers.

wx

U

x1

h0

h1

wx

U

x2

h2

σwx

U

x3

h3

σ

y3sigmoid

I had in

wx

U

x3

h3

σ

mind …

ht=σ(Uwxt)ht-1

σ

RNN–Ihadinmindyourfacts,buddy,nothers.

wx

wh

x1

h0

h1

σwx

wh

x2

h2

σwx

wh

x3

h3

σ

Sigmoid

I had in

wx

wh

x3

h3

σ

mind …

Y=posiVve?Y=negaVve?Finalembeddingrunthroughthesigmoid

funcVon->[0,1]1=posiVve0=negaVveOvenfinalhisusedaswordembeddingforthesentence

UpdatingParametersofanRNN

wx

wh

x1

h0

h1

σwx

wh

x2

h2

σwx

wh

x3

h3

σ

y3sigmoid

Cost

wy

BackpropagaVonthroughVmeGoldlabel=0(negaVve)AdjustweightsusinggradientRepeatmanyVmeswithallexamples

SlidefromRadev

I had in

ProblemwithRNN• Vanishinggradients• BytheVmeweback-propogateallthewaythroughthenetwork,theweightsapproachzero->vanishinggradient• Errorsignals(gradients)inlaterstepsdiminishquicklyanddonotreachearlierinputsignals•  ->Hardtocapturelong-distancedependencies

Whatisalongdistancedependency?• ThestudentswerelisteningtoKathyMcKeownspeak.• Thestudentsin451CSBuildingwerelisteningtoKathyMcKeownspeak

GatedArchitectures• RNN:ateachstateofthearchitecture,theenVrememorystate(h)isreadandwrimen• Gate=binaryvectorgε{0,1}•  Controlsaccesston-dimensionalvectorx�g

• Consider•  Readsentriesfromxspecifiedbyg•  Copiesremainingentriesfroms(orhaswe’vebeenlabelingthehiddenstate)

Example:gatecopiesfromposiVons2and5intheinputRemainingelementscopiedfrommemory

LSTMSolution§ UsememorycelltostoreinformaVonateachVmestep.

§ Use“gates”tocontroltheflowofinformaVonthroughthenetwork.§ Inputgate:protectthecurrentstepfromirrelevantinputs

§ Outputgate:preventthecurrentstepfrompassingirrelevantoutputstolatersteps

§ Forgetgate:limitinformaVonpassedfromonecelltothenext

[slidesfromCatherineFinegan-Dollak]

TransformingRNNtoLSTM

𝑢↓𝑡 =𝜎( 𝑊↓ℎ ℎ↓𝑡−1 + 𝑊↓𝑥 𝑥↓𝑡 )

wx

wh

x1

h0

u1

σ

[slidesfromCatherineFinegan-Dollak]

TransformingRNNtoLSTM

wx

wh

x1

h0

u1

σ

c0

[slidesfromCatherineFinegan-Dollak]

TransformingRNNtoLSTM

𝑐↓𝑡 = 𝑓↓𝑡 ⊙ 𝑐↓𝑡−1 + 𝑖↓𝑡 ⊙ 𝑢↓𝑡 

wx

wh

x1

h0

u1

σ

c0

+ c1

f1

i1

[slidesfromCatherineFinegan-Dollak]

TransformingRNNtoLSTM

𝑐↓𝑡 = 𝑓↓𝑡 ⊙ 𝑐↓𝑡−1 + 𝑖↓𝑡 ⊙ 𝑢↓𝑡 

wx

wh

x1

h0

u1

σ

c0

+ c1

f1

i1

[slidesfromCatherineFinegan-Dollak]

TransformingRNNtoLSTM

𝑐↓𝑡 = 𝑓↓𝑡 ⊙ 𝑐↓𝑡−1 + 𝑖↓𝑡 ⊙ 𝑢↓𝑡 

wx

wh

x1

h0

u1

σ

c0

+ c1

f1

i1

[slidesfromCatherineFinegan-Dollak]

wx

wh

x1

h0

u1

σ

c0

+ c1

f1

i1

TransformingRNNtoLSTM

𝑓↓𝑡 =𝜎( 𝑊↓ℎ𝑓 ℎ↓𝑡−1 + 𝑊↓𝑥𝑓 𝑥↓𝑡 )f1

x1

h0

σwhf

wxf

[slidesfromCatherineFinegan-Dollak]

wx

wh

x1

h0

u1

σ

c0

+ c1

f1

i1

TransformingRNNtoLSTM

𝑖↓𝑡 =𝜎( 𝑊↓ℎ𝑖 ℎ↓𝑡−1 + 𝑊↓𝑥𝑖 𝑥↓𝑡 )

x1

h0

σwhi

wxi

i1

[slidesfromCatherineFinegan-Dollak]

TransformingRNNtoLSTM

wx

wh

x1

h0

u1

σ

c0

+ c1

f1

i1h1

tanh

o1

ℎ↓𝑡 = 𝑜↓𝑡 ⊙ tanh𝑐↓𝑡  

[slidesfromCatherineFinegan-Dollak]

LSTMforSequences

wx

wh

x1

h0

u1

σ

c0

+c1

f1

i1h1

tanh

o1wx

wh

x2

u2

σ

+c2

f2

i2h2

tanh

o2wx

wh

x2

u2

σ

+c2

f2

i2h2

tanh

o2

The cat sat

[slidesfromCatherineFinegan-Dollak]

Bi-LSTM for sentiment!•  Pre-trained Word Embeddings !!

Word Embeddings

..LSTM LSTMLSTMLSTM

p(x)

Feedforward

LSTM

x1

LSTM

x2

LSTM

x3

LSTM

xn

..

Right LSTM Left LSTM

Average

+

SoftmaxSentiment predictions

BiLSTM ..

62

Bi-LSTM enriched with sentiment!word embeddings !!

Sentiment Word Embeddings

..BiLSTM

..

LSTM LSTMLSTMLSTM

p(x)

Feedforward

LSTM

x1

LSTM

x2

LSTM

x3

LSTM

xn

..

Right LSTM Left LSTM

Average

SoftmaxSentiment predictions

Vsentiment(x1)Bilingual Sentiment Scores

Vsentiment(x2) Vsentiment(x3) Vsentiment(xn)

63

RecursiveDeepModelsforSemanticCompositionalityoveraSentimentTreebank•  Socheretal,Stanford2013hmps://nlp.stanford.edu/~socherr/EMNLP2013_RNTN.pdf•  Problemwithpreviouswork:difficultyexpressingthemeaningoflongerphrases• Goal•  TopredictsenVmentatthesentenceorphraselevel•  CaptureeffectofnegaVonandconjuncVons•  SenVmentTreebank•  RecursiveNeuralTensorNetwork

SentimentTreebank• Moviereviewexcerptsfromromentomatoes.com(Pang&Lee2005)•  10,662sentences• ParsedbyStanfordParser(Klein&Manning2003)•  215,154phrases• EachphraselabeledforsenVmentusingAmazonMechanicalTurk(AMT)•  5classesemerge:negaVve,somewhatnegaVve,neutral,somewhatposiVve,posiVve

Example--verynegaVve++veryposiVve-  NegaVve+posiVve0neutral

RecursiveNeuralModels

RNN:RecursiveNeuralNetwork

WaretheweightstolearnWεf=tanh

MV-RNNMatrixvectorRNN• Introduceweightmatrixassociatedwitheachnon-terminal(P2foradjP)andterminal(Afora)• a=not,b=very,c=good

RNTN:RecursiveNeuralTensorNetwork•  TheMV-RNNhastoomanyparameterstolearn(sizeofvocabulary)•  CanwegetcomposiVonalitywithreducedparameters?•  P1=f([ab]u1u2a)u3u4b=f([ab]u1a+u2b)u3a+u4b

=f(u1aa+u2ab+u3ab+u4bb)

Results

Positive–“mostcompelling”

Negative–“leastcompelling”

HandlingConjunctions