CS4705NaturalLanguageProcessingFall2017

ProfessorKathyMcKeown

1

ClassSizeandIn-classDiscussion•  Piazzatointeractinclass.Forin-classandvideostudents•  Regularintervals:pausetoanswerques@ons•  Iwillalsoaskques@onsofyou

•  Answerswillbeshownontheslide•  Makingyournameshowntofellowstudents•  Ontheinstructorsystems,wewillseewhoposedaques@on->assigncreditforinterac@on

•  Nopenaltyforwronganswerexceptforinappropriatecontent

•  Mayuseaninteres@nganswertofurtherdiscussion•  In-classpar@cipa@onwillcounttowardsyourgrade 2

DownloadPiazzaApponyourPhone

3

ClassPolicyonElectronics• CellphoneinclassOKandneededforPiazzainterac@on• Keeplaptopsclosedordon’tbringtoclass

4

Today• WhatisNLP?• ClassLogis@cs• Whatwillwecover• Helpfulbackground• Classhomeworkandexams

5

Whatwillwestudyinthisclass• Howcanmachinesunderstandandgeneratelanguage?•  Examplesdrawnfromnaturallyoccurringcorpora•  Theoriesaboutlanguage•  Algorithms

•  Sta@s@calmethods•  Applica@ons

6

KnowledgeNeeded• Morphology:wordforma@on• Syntax:wordorder• Seman@cs:wordmeaningandcomposi@on• Pragma@cs:Influenceofcontextandsitua@on

Goal:discoverwhatthespeakermeant 7

Morphology•  Importantforsearch,machinetransla@on,summariza@on• UnionAc9vi9esinNewYork•  Singular/plural•  Union/unions•  Ac@vity/ac@vi@es

•  Otherlanguagesaremorphologicallyrich•  Arabic:definiteembeddedintheword(cli@cs):Theunion(Al+)vs.aunion,unions•  German:casepartoftheword(subjvsobj)

•  Arethereexamplesinyourlanguage? 8

Responses

9

Newsarticletitles•  Stud@resout•  Eyedropsoffshelf•  Teacherstrikesidlekids• Drunkgetsninemonthsinviolincase•  Enragedcowinjuresfarmerwithax•  BanonnudedancingonGovernor’sdesk• Hospitalsaresuedbysevenfootdoctors•  Redtapeholdsupnewbridges• Governmentheadseeksarms•  Pa@entatdeath’sdoor–doctorspullhimthrough•  InAmericaawomanhasababyevery15minutes 10

Syntax• Partofspeechtagging:isawordanoun,verb,adverb,adjec@ve,etc?• Parsing•  Iden@fyingcons@tuents•  NP:KathyMcKeown,amaninthepark•  VP:waslookingup,hadrisen

•  Iden@fyingsubjectsandobjects•  BillhitJohnvsJohnhitBill

• Modifica@on•  Johnsawthemanintheparkwithatelescope 11

PartofSpeechtagging• Stud9resout•  Tires:anounoraverb?• Eyedropsoffshelf• Drops:anounoraverb?• Teacherstrikesidlekids•  Strikes:anounoraverb?

12

Responses

13

ConstituentStructureandModiKication• TheproblemofPPakachmentEnragedcowinjuresfarmerwithax• [Enragedcow]injuresfarmer[withax]• [Enragedcow]injuresfarmer[withax]

14

RepresentingmodiKicationwithbrackets• [Enragedcow][injures[farmer[withax]]]• [Enragedcow][injures[farmer][withax]]]

15

ConstituentStructureandModiKication• TheproblemofPPakachmentBanonnudedancingongovernor’sdesk• [Ban]on[nudedancing][ongovernor’sdesk]• Therearetwopossiblemodifica@ons?Whatarethey?• Whichoneiscorrect?

16

Response

17

ConstituentStructureandModiKication

• TheproblemofPPakachmentBanonnudedancingongovernor’sdesk• [[Ban]on[nudedancing]][ongovernor’sdesk]• [Ban]on[[nudedancing][ongovernor’sdesk]]

18

NounnounmodiKication• Waterfountain:afountainthatsupplieswater• Waterballet:aballetthattakesplaceinwater• Watermeter:adevice(calledameter)thatmeasureswater• Waterbarometer:abarometerthatuseswater(insteadofmercury)tomeasureairpressure• Waterglass:aglassthatismeanttoholdwater

19

NounnounmodiKication:constituentstructure• Whichcons9tuentstructurebestrepresentsthemeaningofcountrysongpla9numalbum?

1.  [country[song[pla9numalbum]]]2.  [country[[songpla9num]album]]3.  [[countrysong][pla9numalbum]]4.  [[country[songpla9num]]album]5.  [[[countrysong]pla9num]album]

20

Response

21

NounnounmodiKicationandheadlines• Hospitalsaresuedbysevenfootdoctors

• Hospitalsaresuedby[[sevenfoot]doctors]

• Hospitalsaresuedby[seven[footdoctors]]

22

WordMeaning• Redtapeholdsupnewbridges• Holdsup:

1.  [TRANSITIVE]tosupportsomeoneorsomethingsothattheydonotfalldown•  Herlegswerealmosttooshakytoholdherup.

2.  [TRANSITIVE][OFTENPASSIVE]tocauseadelayforsomeoneorsomething,ortomakethemlate•  SorryI’mlate,butmyflightwasheldup.

• Governmentheadseeksarms• Head:• Arms: 23

Pragmatics• Discoursecontext•  Johnwenttothestore.HeboughtbreadandbuYer.

• Situa@onal(realworld)context• DayanertheCharlokesville,VAriots• Hisirresponsibleac9onstookthelifeofayoungwomanwhowasjustbeginningheradultlife.

• Commonsenseknowledge• Bostoncalledandle[amessageforJoe. 24

TheLanguageofGenres:news,journal,socialmedia,novel?

•  Devasta@ng,butyetamazingstorm.IMBYjustsomebranches,tonofleaves,…ThissurpassesanyBigdaddystorm,shouldbecalledBigMama.(ComputerGuy)

•  Producedbyateamof26scien@stsledbytheUniversityofNewSouthWalesClimateResearchCentre,theDiagnosisconvincinglyprovesthattheeffectsofglobalwarminghavegokenworseinthelastthreeyears.(Somervilleetal2011)

•  HurricaneSandychurnedabout290milesofftheMid-Atlan@ccoastSundaynight,withtheNa@onalHurricaneCenterrepor@ngthatthemonsterstormwasexpectedtocomeashorewithnear-hurricane-forcewindsandpoten@ally"life-threatening"stormsurgeflooding.

•  WhatisthemaYer?'Icried.'Awreck!Closeby!'

25

Machinelearningframework• Data(onenlabeled)• Extrac@onof“features”fromtextdata• Predic@onofoutput

26

Machinelearningframework• Data(onenlabeled)• Extrac@onof“features”fromtextdata• Predic@onofoutputWhatdataisavailableforlearning?

27

Machinelearningframework• Data(onenlabeled)• Extrac@onof“features”fromtextdata• Predic@onofoutputWhatfeaturesyieldgoodpredic9ons?

28

MachineLearningMethods• Supervised•  Supportvectormachine,NaïveBayes,Logis@cregression•  Sequencelabeling:HiddenMarkovModeling(HMM),Condi@onalRandomFields(CRF)•  Neuralnetworks

• Unsupervised•  Clustering

• Semi-supervised•  Boot-strapping,self-training,co-training•  Distantlearning 29

Wheredoesthedatacomefrom?• Manuallylabeled• Naturallyoccurring• Anoisy,butplen@fulsubs@tute

30

CoreNLP• Morphologicalanalysis• Partofspeechtagging• Parsing

31

Applications• Searchingverylargetextandspeechcorpora•  E.g.,theweb

• Ques@onansweringovertheweb• Transla@ngbetweenonelanguageandanother:e.g.,ChineseandEnglish• Summarizingtext:e.g.,youremail,thenews,reviews• Sen@mentanalysis• Genera@ngtexts• Dialogsystems:Amtrak’sJulie

32

Logistics

33

Instructor• KathyMcKeown•  Office:722CEPSR•  NLPGroup•  35yearsatColumbia,FoundingDirectoroftheDataScienceIns@tute(juststeppeddown)

•  Research•  Summariza@on•  Ques@onAnswering•  LanguageGenera@on•  Sen@mentanalysis• Mul@lingualapplica@ons 34

TAs• ElsbethTurcan(headTA)• DheerajKalmekolan• ApoorvKulshreshtha• RobertKwiatkowski• Fei-TzinLee• BhavanaRamachandra• SamarthTripathi

35

Background• Programming.WewillusePython• Inaddi@on,atleastone:• Ar@ficialIntelligence• Machinelearning• ProgrammingLanguagesandTranslators•  Sta@s@cs 36

Syllabus• Availableat:hkp://www.cs.columbia.edu/~kathy/NLP/2017

37

Textbooks• SpeechandLanguageProcessing,2ndEdi@on,byJurafskyandMar@n.ItwillbeavailablefromBookCulture,aswellasfromAmazonandotheronlineproviders.ItisalsoonreserveintheScienceLibrary.

• NeuralNetworkMethodsforNaturalLanguageProcessingbyYoavGoldberg.Itisavailableonlinebutyoucanalsopurchasehardcopyfromthepublisher. 38

Assignments• 4homeworkassignments:3programming,1wriken

•  WewillbeusingGoogleCloud•  HW0:•  Worth2points,butifyoudonot/cannotdoit,thisisnottheclassforyou.

•  Setsupyourgooglecloudaccountproperly•  Fourfreelatedays•  Anerthat10%offforeachdaylate

• Midtermandfinal• Evalua@on:50%homework+40%exams+10%classpar@cipa@on(viaPiazza)

39

AcademicIntegrity•  Copyingorparaphrasingsomeone'swork(codeincluded),orpermi{ngyourownworktobecopiedorparaphrased,evenifonlyinpart,isforbidden,andwillresultinanautoma@cgradeof0fortheen@reassignmentorexaminwhichthecopyingorparaphrasingwasdone.Yourgradeshouldreflectyourownwork.Ifyouaregoingtohavetroublecomple@nganassignment,talktotheinstructororTAinadvanceoftheduedateplease.Everyone:Read/writeprotectyourhomeworkfilesatall@mes.

40

ForNextClass• ReadChapters1-2ofJ&M,Chapter1ofNN• Ques@ons?(UsePiazza)

41

Questions

42