Upload
others
View
3
Download
0
Embed Size (px)
Citation preview
SentimentAnalysis
Announcements• HW1isduetodayat11:59pm• HW2willcomeoutMonday• IfyouuselatedaysforHW1,MUSTSTATEITINYOURSUBMISSION.• Reading:Today:C4.2NLP• Monday:C8.1-8.3SpeechandLanguage,
8.1NLP• RecommendreadingchaptersinYoavGoldbergonneuralnetsandJurafskyandMarVnaswell.• BotharemoreintuiVve
Today• SenVmentanalysistasks:definiVon• SenVmentresources• TradiVonalsupervisedapproach• Neuralnetapproach
Whatissentiment?• ExpressionofposiVveornegaVveopinions• ..Towardsatopic,person,event,enVty• ..Towardsanaspect(e.g.,service,foodorambienceinarestaurant)
Whysentimentanalysis?• SenVmentiscommoninonlinepla[orms• Peoplewriteabouttheirpersonalviewpoints
• UsefultounderstandwhatpeoplethinkaboutpoliVcalissues,poliVcalcandidates,importanteventsoftheday• UsefulforgeneraVngsummariesofreviews:restaurants,products,movies
Thesentimentanalysistask(s)• SubjecVvevsobjecVve• Polarity:PosiVve,negaVveorneutral• DowehavesenVmenttowardsatarget?OraspectbasedsenVment?• What/whoisthesenVmentsource?
SubjectivevsObjective• �Atseveraldifferentlayers,it’safascina3ngtale.[“Who’sSpyingonOurComputers”,GeorgeMelloanWallStJournal.(Bookreview)• BellIndustriesIncincreaseditsquarterlyto10centsfrom7centsashare.
ExamplesfromWeibeetal2004
Positive/Negative/Neutral• FromUseNet:
• NegaVve:Ihadinmindyourfacts,Buddy,nothers.• PosiVve:Nicetouch.“Alleges”whateverfactspostedarenotinyourpersonaofwhatis“real”• Neutral:Marchappearstobeanes3matewhileearlieradmissioncannotbeen3relyruledout,"accordingtoChen,alsoTaiwan'schiefWTOnego3ator
ExamplesfromWeibeetal2004andRosenthal2014
SubjectivePhrases• TheforeignministrysaidThursdaythatitwas“surprised,toputitmildly”bytheU.S.StateDepartment’scri0cismofRussia’shumanrightsrecordandobjectedinparVculartothe“odious”secVononChechnya.[MoscowTimes,03/08/2002]• Subjec3vityanalysisiden3fiestextthatrevealsanauthor’sthoughts,beliefsorotherprivatestates.
ExamplesfromWeibeetal2004
SubjectivePhrasesandSources• TheforeignministrysaidThursdaythatitwas“surprised,toputitmildly”bytheU.S.StateDepartment’scri0cismofRussia’shumanrightsrecordandobjectedinparVculartothe“odious”secVononChechnya.[MoscowTimes,03/08/2002]• Whowassurprised?• WhowascriVcal?
ExamplesfromWeibeetal2004
SentimenttowardsTarget• IpreUymuchenjoyedthewholemovie.Target=wholemovie,senVment=posiVve.• Bulgariaiscri3cizedbytheEUbecauseofslowreformsinthejudiciarybranch,thenewspapernotes.Target=Bulgaria,senVment=negaVve• Stanishevwaselectedprimeministerin2005.Sincethen,hehasbeenaprominentsupporterofhiscountry’saccessiontotheEU.Target=country’saccesstotheEU,senVment=posiVve
ExamplesfromBreck&Cardie
Datasets(Sem-evaldatasetsalsoused)
2000sentencesineachcorpus
12
Corpus AverageWordCount
AverageCharacterCount
Subjec6vePhrases
Objec6vePhrases
VocabularySize
CharacterLengthRestric6ons
LiveJournal 14.67 66.47 3035(39%) 4747(61%) 4747 30-120
MPQA 31.64 176.68 3325(41%) 4754(59%) 7614 none
Twimer 25.22 118.55 2091(36%) 3640(64%) 8385 0-140
Wikipedia 15.57 77.20 2643(37%) 4496(63%) 4342 30-120
MPQA:extensivelyannotateddatasetbyStoyanav,CardieandWeibe2004.15opinionorientedqusVons,15factorientedquesVons.Alongwithtextspansfrom252arVcles.
RosenthalandMcKeown2013)
ExampleSentences
LiveJournal iwillhavetosVcktomycanonfilmslrunVlinafewyearsicanaffordtoupgradeagain:)
MPQA ThesaleinfuriatedBeijingwhichregardsTaiwananintegralpartofitsterritoryawaiVngreunificaVon,byforceifnecessary.
Twimer RT@tashjade:That’sreallysad,CharlieRT“UnVltonightIneverrealisedhowfuckedupIwas”-CharlieSheen#sheenroast
Wikipedia PerhapsifreportedcriVcallybyawesternsourcebutcertainlynotbyanIsraelisource.
13
SubjecVve ObjecVve
SentimentLexicons• GeneralInquirer• SenVWordNet• DicVonaryofAffect(DAL)
DictionaryofAffectinLanguage• DicVonaryof8742wordsbuilttomeasuretheemoVonalmeaningoftexts• Eachwordisgiventhreescores(scaleof1to3)• pleasantness-alsocalledevaluaVon(ee)• acVveness(aa)• andimagery(ii)
C.M.Whissel.1989.Thedic6onaryofaffectinlanguage.InR.PlutchikandH.Kellerman,editors,EmoVon:theoryresearchandexperience,volume4,London.Acad.Press.
15
Emoticons• 1000emoVconsweregatheredfromseverallistsavailableontheinternet• Wekeptthe192emoVconsthatappearedatleastonceandmappedeachemoVcontoasingleworddefiniVon
16
Methods• Pre-processingsteps• EmoVconkeysandcontracVonexpansion• Chunkerandtagger*• LexicalFeatures*• SyntacVcFeatures*• SocialMediaFeatures
*ApoorvAgarwal,FadiBiadsy,andKathleenR.McKeown.2009.Contextualphrase-levelpolarityanalysisusinglexicalaffectscoringandsyntac6cn-grams.InProceedingsofEACL’09
17
PreprocessingLiveJournal [i]/NPsub[willhavetos6ck]/VPobj[to]/PPobj[mycanonfilmslr]/NPobj[unVl]/
PPobj[in]/PPobj[afewyears]/NPsub[i]/NPsub[canaffordtoupgrade]/VPobj[again:)]/NPsub
MPQA [Thesale]/NPsub[infuriated]/VPobj[Beijing]/NPobj[which]/NPsub[regards]/VPsub[Taiwan]/NPobj[anintegralpart]/NPsub[of]/PPobj[itsterritoryawaiVngreunificaVon,]/NPobj[by]/PPobj[force]/NPsub[if]/obj[necessary.]/sub
Twimer [RT@tashjade:]/NPobj [That]/Npobj[is]/VPsub[really]/sub[sad,]/sub[CharlieRT]/NPobj[”]/NPobj[UnVl]/PPobj[tonight]/NPsub[I]/NPsub[never]/sub[realised]/VPsub[how]/sub[fucked]/VPsub[up]/PPobj[I]/NPsub[was]/VPsub[”]/obj[-CharlieSheen#sheenroast]/NPobj
Wikipedia [Perhaps]/sub[if]/obj[reported]/VPsub[cri6cally]/sub[by]/PPobj[awesternsourcebut]/NPsub[certainlynot]/sub[by]/PPobj[anIsraelisource.]/NPsub
18Xuan-HieuPhan,CRFChunker:CRFEnglishPhraseChunkerhmp://crfchunker.sourceforge.net/,2006
LexicalFeatures• POSTags*• N-grams*• Performedchi-squarefeatureselecVononthen-grams
*ApoorvAgarwal,FadiBiadsy,andKathleenR.McKeown.2009.Contextualphrase-levelpolarityanalysisusinglexicalaffectscoringandsyntac6cn-grams.InProceedingsofEACL’09
19
SyntacticFeatures• Usethemarkedupchunkstoextractthefollowing:*• n-grams:1-3words• POS:NP,VP,PP,JJ,other• PosiVon:target,right,lev• SubjecVvity:subjecVve,objecVve• Minandmaxpleasantness
*ApoorvAgarwal,FadiBiadsy,andKathleenR.McKeown.2009.Contextualphrase-levelpolarityanalysisusinglexicalaffectscoringandsyntac6cn-grams.InProceedingsofEACL’09
20
SocialMediaFeatures
21
SingleCorpusClassiLication
BalancedUnbalanced
• LogisVcRegressioni
• 10runsof10-foldcross-validaVon
• StaVsVcalsignificanceusingthet-testwithp=.001
22
Experiment LiveJournal MPQA Twimer Wikipedia
n-gramsize 100 2000 none none
majority 58% 59% 64% 63%
JustDAL 76.5% 75.7% 83.6% 80.4%
DicVonaries+SM 77.1% 76.1% 84% 81.4%
Wordnet 76.7% 75.6% 84% 80.7%
Wordnet+SM 77.1% 76.1% 84.2% 81.4%
DicVonaries 76.6% 75.7% 83.9% 80.7%
SM 77% 76.1% 83.7% 81.2%
Experiment LiveJournal MPQA Twimer Wikipedia
n-gramsize 100 200 none none
majority 50% 50% 50% 50%
JustDAL 74.7% 75.7% 81.9% 79.3%
DicVonaries+SM 76.7% 76.2% 82.6% 80.2%
Wordnet 75.1% 75.8% 82.4% 79.1%
Wordnet+SM 76.6% 75.3% 82.6% 80.3%
DicVonaries 75.3% 75.8% 82.4% 79.1%
SM 76.2% 76.3% 82.2% 80.4%
SocialMediaErrorAnalysis• Wikipedia• PunctuaVonwasusefulasafeaturefordeterminingthataphraseisobjecVveifitisasmallphrase.However,severalsubjecVvephraseswereincorrectlyclassifiedbecauseofthis 23
0%
20%
40%
60%
80%
100%
Wikipedia
subjecVveobjecVve
SocialMediaErrorAnalysis• Twimer• EllipseshelpindicatethatasentenceisobjecVve.Theaccuracyimprovedfrom82%to92%forsentenceswiththisfeature• AllothersocialmediafeatureswereincorrectlyclassifiedasobjecVve/subjecVvedependingonthesocialmediapreference. 24
0%
20%
40%
60%
80%
100%
TwiZer
subjecVveobjecVve
SocialMediaErrorAnalysis• LiveJournal• OutofVocabularywordsandpunctuaVonwerethemostusefulsocialmediafeatures.• InalldatasetsthepunctuaVonfeaturecausedcloseto50/50exchangebutthefeaturewasbestinLiveJournal. 25
0%
20%
40%
60%
80%
100%
Livejournal
subjecVveobjecVve
NeuralNetworkApproachestoSentiment• TakeastandardRNN• Takealabeleddataset(e.g.,IMDBsenVmentdataset)• IniValizewithpre-trainedwordembeddings(wordtovecorglove)• UsesigmoidtopredictbinarysenVmentlabels:posiVvevsnegaVve.
Languageismadeupofsequences• Sofarwehaveseenembeddingsforwords• (andmethodsforcombiningthroughvectorconcatenaVonandarithmeVc)
• Buthowcanweaccountforsequences?• Wordsassequencesoflemers• Sentencesassequencesofwords• Documentsassequencesofsentences
RecurrentNeuralNetworks• Representarbitrarilysizedsequencesinfixed-sizevector• GoodatcapturingstaVsVcalregulariVesinsequences(ordermamers)• IncludesimpleRNNs,Longshort-termmemory(LSTMs),GatedRecurrentUnit(GRUs)
[Thangetal.2013] [Bowmanetal.2014]
Learningwordmeaning Logicalentailmentfromtheirmorphsusingcompositional
semanticsviaRNNs
MachineTranslation(Sequences)
• Sequence-to-sequence• Sutskeveretal.2014
RNNAbstraction• RNNisafuncVonthattakesanarbitrarylengthsequenceasinputandreturnsasingledoutdimensionalvectorasoutput• Input:x1:n=x1x2…xn(xiεRd-in)• Output:ynεRd-out
OOutputvectoryusedforfurtherpredicVon
RNNCharacteristics• CancondiVonontheenVresequencewithoutresorVngtotheMarkovassumpVon• Cangetverygoodlanguagemodelsaswellasgoodperformanceonmanyothertasks
RNNsaredeLinedrecursively• BymeansofafuncVonRtakingasinputastatevectorhi-1andaninputvectorxi• Returnsanewstatevectorhi• ThestatevectorcanbemappedtoanoutputvectoryiusingasimpledeterminisVcfuncVon• AndfedthroughsovmaxforclassificaVon.
wx
wh
RecurrentNeuralNetworks
x1
h0
h1
ℎ↓𝑡 =𝜎( 𝑊↓ℎ ℎ↓𝑡−1 + 𝑊↓𝑥 𝑥↓𝑡 )
σ
SlidefromRadev
wx
wh
RNN
x1
h0
h1
ℎ↓𝑡 =𝜎(𝑊↓ℎ ℎ↓𝑡−1 + 𝑊↓𝑥 𝑥↓𝑡 )𝑦↓𝑡 =𝑠𝑜𝑓𝑡𝑚𝑎𝑥( 𝑊↓𝑦 ℎ↓𝑡 )
σ
y1so[maxwy
SlidefromRadev
RNN
wx
wh
x1
h0
h1
σwx
wh
x2
h2
σwx
wh
x3
h3
σ
y3so[max
The cat sat
wy
SlidefromRadev
UpdatingParametersofanRNN
wx
wh
x1
h0
h1
σwx
wh
x2
h2
σwx
wh
x3
h3
σ
y3so[max
The cat sat
Cost
wy
BackpropagaVonthroughVme
SlidefromRadev
Example• Foreachsentenceinthetrainingcorpus,classify,comparetogoldstandardandcomputeloss,backpropagate.• Recallthatwemayusemini-batchessothatwe’renotback-propagaVngforeachexample
• Ihadinmindyourfacts,Buddy,nothers.
RNN–Ihadinmindyourfacts,buddy,nothers.
wx
wh
x1
h0
h1
σwx
wh
x2
h2
σwx
wh
x3
h3
σ
y3sigmoid
I had in
wx
wh
x3
h3
σ
mind …
Inthisoverview,wreferstotheweightsButtherearedifferentkindsofweightsLet’sbemorespecific
RNN–Ihadinmindyourfacts,buddy,nothers.
wx
U
x1
h0
h1
wx
U
x2
h2
σwx
U
x3
h3
σ
y3sigmoid
I had in
wx
U
x3
h3
σ
mind …
σ
RNN–Ihadinmindyourfacts,buddy,nothers.
wx
U
x1
h0
h1
wx
U
x2
h2
σwx
U
x3
h3
σ
y3sigmoid
I had in
wx
U
x3
h3
σ
mind …
σ
RNN–Ihadinmindyourfacts,buddy,nothers.
wx
U
x1
h0
h1
wx
U
x2
h2
σwx
U
x3
h3
σ
y3sigmoid
I had in
wx
U
x3
h3
σ
mind …
ht=σ(Uwxt)ht-1
σ
RNN–Ihadinmindyourfacts,buddy,nothers.
wx
wh
x1
h0
h1
σwx
wh
x2
h2
σwx
wh
x3
h3
σ
Sigmoid
I had in
wx
wh
x3
h3
σ
mind …
Y=posiVve?Y=negaVve?Finalembeddingrunthroughthesigmoid
funcVon->[0,1]1=posiVve0=negaVveOvenfinalhisusedaswordembeddingforthesentence
UpdatingParametersofanRNN
wx
wh
x1
h0
h1
σwx
wh
x2
h2
σwx
wh
x3
h3
σ
y3sigmoid
Cost
wy
BackpropagaVonthroughVmeGoldlabel=0(negaVve)AdjustweightsusinggradientRepeatmanyVmeswithallexamples
SlidefromRadev
I had in
ProblemwithRNN• Vanishinggradients• BytheVmeweback-propogateallthewaythroughthenetwork,theweightsapproachzero->vanishinggradient• Errorsignals(gradients)inlaterstepsdiminishquicklyanddonotreachearlierinputsignals• ->Hardtocapturelong-distancedependencies
Whatisalongdistancedependency?• ThestudentswerelisteningtoKathyMcKeownspeak.• Thestudentsin451CSBuildingwerelisteningtoKathyMcKeownspeak
GatedArchitectures• RNN:ateachstateofthearchitecture,theenVrememorystate(h)isreadandwrimen• Gate=binaryvectorgε{0,1}• Controlsaccesston-dimensionalvectorx�g
• Consider• Readsentriesfromxspecifiedbyg• Copiesremainingentriesfroms(orhaswe’vebeenlabelingthehiddenstate)
Example:gatecopiesfromposiVons2and5intheinputRemainingelementscopiedfrommemory
LSTMSolution§ UsememorycelltostoreinformaVonateachVmestep.
§ Use“gates”tocontroltheflowofinformaVonthroughthenetwork.§ Inputgate:protectthecurrentstepfromirrelevantinputs
§ Outputgate:preventthecurrentstepfrompassingirrelevantoutputstolatersteps
§ Forgetgate:limitinformaVonpassedfromonecelltothenext
[slidesfromCatherineFinegan-Dollak]
TransformingRNNtoLSTM
𝑢↓𝑡 =𝜎( 𝑊↓ℎ ℎ↓𝑡−1 + 𝑊↓𝑥 𝑥↓𝑡 )
wx
wh
x1
h0
u1
σ
[slidesfromCatherineFinegan-Dollak]
TransformingRNNtoLSTM
wx
wh
x1
h0
u1
σ
c0
[slidesfromCatherineFinegan-Dollak]
TransformingRNNtoLSTM
𝑐↓𝑡 = 𝑓↓𝑡 ⊙ 𝑐↓𝑡−1 + 𝑖↓𝑡 ⊙ 𝑢↓𝑡
wx
wh
x1
h0
u1
σ
c0
+ c1
f1
i1
[slidesfromCatherineFinegan-Dollak]
TransformingRNNtoLSTM
𝑐↓𝑡 = 𝑓↓𝑡 ⊙ 𝑐↓𝑡−1 + 𝑖↓𝑡 ⊙ 𝑢↓𝑡
wx
wh
x1
h0
u1
σ
c0
+ c1
f1
i1
[slidesfromCatherineFinegan-Dollak]
TransformingRNNtoLSTM
𝑐↓𝑡 = 𝑓↓𝑡 ⊙ 𝑐↓𝑡−1 + 𝑖↓𝑡 ⊙ 𝑢↓𝑡
wx
wh
x1
h0
u1
σ
c0
+ c1
f1
i1
[slidesfromCatherineFinegan-Dollak]
wx
wh
x1
h0
u1
σ
c0
+ c1
f1
i1
TransformingRNNtoLSTM
𝑓↓𝑡 =𝜎( 𝑊↓ℎ𝑓 ℎ↓𝑡−1 + 𝑊↓𝑥𝑓 𝑥↓𝑡 )f1
x1
h0
σwhf
wxf
[slidesfromCatherineFinegan-Dollak]
wx
wh
x1
h0
u1
σ
c0
+ c1
f1
i1
TransformingRNNtoLSTM
𝑖↓𝑡 =𝜎( 𝑊↓ℎ𝑖 ℎ↓𝑡−1 + 𝑊↓𝑥𝑖 𝑥↓𝑡 )
x1
h0
σwhi
wxi
i1
[slidesfromCatherineFinegan-Dollak]
TransformingRNNtoLSTM
wx
wh
x1
h0
u1
σ
c0
+ c1
f1
i1h1
tanh
o1
ℎ↓𝑡 = 𝑜↓𝑡 ⊙ tanh𝑐↓𝑡
[slidesfromCatherineFinegan-Dollak]
LSTMforSequences
wx
wh
x1
h0
u1
σ
c0
+c1
f1
i1h1
tanh
o1wx
wh
x2
u2
σ
+c2
f2
i2h2
tanh
o2wx
wh
x2
u2
σ
+c2
f2
i2h2
tanh
o2
The cat sat
[slidesfromCatherineFinegan-Dollak]
Bi-LSTM for sentiment!• Pre-trained Word Embeddings !!
Word Embeddings
..LSTM LSTMLSTMLSTM
p(x)
Feedforward
LSTM
x1
LSTM
x2
LSTM
x3
LSTM
xn
..
Right LSTM Left LSTM
Average
+
SoftmaxSentiment predictions
BiLSTM ..
62
Bi-LSTM enriched with sentiment!word embeddings !!
Sentiment Word Embeddings
..BiLSTM
..
LSTM LSTMLSTMLSTM
p(x)
Feedforward
LSTM
x1
LSTM
x2
LSTM
x3
LSTM
xn
..
Right LSTM Left LSTM
Average
SoftmaxSentiment predictions
Vsentiment(x1)Bilingual Sentiment Scores
Vsentiment(x2) Vsentiment(x3) Vsentiment(xn)
63
RecursiveDeepModelsforSemanticCompositionalityoveraSentimentTreebank• Socheretal,Stanford2013hmps://nlp.stanford.edu/~socherr/EMNLP2013_RNTN.pdf• Problemwithpreviouswork:difficultyexpressingthemeaningoflongerphrases• Goal• TopredictsenVmentatthesentenceorphraselevel• CaptureeffectofnegaVonandconjuncVons• SenVmentTreebank• RecursiveNeuralTensorNetwork
SentimentTreebank• Moviereviewexcerptsfromromentomatoes.com(Pang&Lee2005)• 10,662sentences• ParsedbyStanfordParser(Klein&Manning2003)• 215,154phrases• EachphraselabeledforsenVmentusingAmazonMechanicalTurk(AMT)• 5classesemerge:negaVve,somewhatnegaVve,neutral,somewhatposiVve,posiVve
Example--verynegaVve++veryposiVve- NegaVve+posiVve0neutral
RecursiveNeuralModels
RNN:RecursiveNeuralNetwork
WaretheweightstolearnWεf=tanh
MV-RNNMatrixvectorRNN• Introduceweightmatrixassociatedwitheachnon-terminal(P2foradjP)andterminal(Afora)• a=not,b=very,c=good
RNTN:RecursiveNeuralTensorNetwork• TheMV-RNNhastoomanyparameterstolearn(sizeofvocabulary)• CanwegetcomposiVonalitywithreducedparameters?• P1=f([ab]u1u2a)u3u4b=f([ab]u1a+u2b)u3a+u4b
=f(u1aa+u2ab+u3ab+u4bb)
Results
Positive–“mostcompelling”
Negative–“leastcompelling”
HandlingConjunctions