Natural Language Processing

Yoav GoldbergComputer Science Department

Presented in Academic Writing in English course

Please try and make your own

presentations for dummies as well

What is a Natural Language?

• Natural Languages are languages of humans(such as Hebrew, English, Arabic, Hindi, Latin..)

• These can be either written or spoken

What is Natural Language Processing?

• A subfield of Computer Science

What is Natural Language Processing?

• A subfield of Computer Science

• 20-30 years ago:

What is Natural Language Processing?

• A subfield of Computer Science

• 20-30 years ago:

“NLP is about making the computer understand natural language”

What is Natural Language Processing?

• A subfield of Computer Science

• 20-30 years ago:

“NLP is about making the computer understand natural language”

But today we know that:

• Language is HARD

• Computers are STUPID

What is Natural Language Processing?

• A subfield of Computer Science• 20-30 years ago:

“NLP is about making the computer understand natural language”

But today we know that:• Language is HARD• Computers are STUPID

The computer will never understand language

Language is Hard

הרכבת

Language is Hard

הרכבת

הרכבת המהירה לחיפה

הרכבת הממשלה

הרכבת את הפאזל

הרכבת על הסוס?

Language is Hard

כפיות

Language is Hard

• I play the Bass

• I hate banks

Language is Hard

• I play the Bass

• Some people like Bass fishing

• I hate banks

• River banks are fun places

Some of the next ones might be hard also for humans!

Language is Hard

• Thin people eat candy

Language is Hard

• Thin people eat candy

• Fat people eat candy

Language is Hard

• Thin people eat candy

• Fat people eat candy

• Fat people eat steaks

Language is Hard

• Thin people eat candy

• Fat people eat candy

• Fat people eat steaks

• Fat people eat accumulates

Language is Hard

• Flying planes are dangerous

• Flying planes is dengerous

Language is Hard

• I saw a man on the hill with a telescope

Language is Hard

• I saw a man on the hill with a telescope

Who has the telescope?

Who is on the hill?

Language is Hard

• I saw a man on the hill with a telescope

Who has the telescope?Who is on the hill?

(and this takes for granted that the sentence is not about a very cruel way of killing someone)

Ok, so I hope I convinved you language is hard

And these examples didn’t even touch the subject of what understanding is all about!

So computers will never understand language

So computers will never understand language

How will I ever finish my thesis??

Fortunately, we can go a long way by cheating

So what is Natural Language Processing today?

• Natural language processing is about making computer programs that can do seemingly intelligent things with Natural Language input

• Or, in other words, finding devious ways of cheating people to think the computer can understand language to some extent

Lies, Damn Lies, and Statistics

• One of our main cheating tools is Statistics

• Let me demonstrate

For example:

• Humans know that:I have a spelling checker

• Makes far more sense than: Eye halve a spelling chequer

Can computers do that?

Example (cont.)

• “Make Sense” is hard, but we can cheat by changing the question:

Which of the following is More Probable?

“I have a spelling checker”

“Eye halve a spelling chequer”

Example (cont.)

• This is still hard, but we can cheat yet again by asking several easier questions

What’s the probability of seeing:

halve after Eye ?

a after halve ?

spelling after a ?

chequer after spelling ?

have after I ?a after have ? spelling after a ? checker after spelling ?

Example (cont.)

• This is still hard, but we can cheat yet again by asking several easier questions

What’s the probability of seeing:

halve after Eye ?

a after halve ?

spelling after a ?

chequer after spelling ?

have after I ?a after have ? spelling after a ? checker after spelling ?

(we are assuming every words depends only on the word preceding it. This is ofcourse wrong.)

Example (cont.)

Seeing halve after Eye:

P(halve | Eye)

Example (cont.)

Seeing halve after Eye:

P(halve | Eye)

= count(Eye halve) / count(Eye)

Example (cont.)

Seeing halve after Eye:

P(halve | Eye)

= count(Eye halve) / count(Eye)

= 14,600 / 301,000,000

= 4.85e-5

Example (cont.)

Seeing halve after Eye:P(halve | Eye) = count(Eye halve) / count(Eye) = 14,600 / 301,000,000

= 4.85e-5In the same manner:P( a | halve ) = 0.0033 P( have | I ) = 0.19P( spelling | a) = 1.5e-4 P ( a | have ) = 0.45P( chequer | spelling ) = 2.55e-4 P ( checker | spelling ) = 0.012

Example (cont)

• Combining the probabilities, we can estimate:

P(“Eye halve a spelling chequer”)

P(“I have a spelling checker”)

Example (cont)

• Combining the probabilities, we can estimate:

P(“Eye halve a spelling chequer”) 6.12e-15

P(“I have a spelling checker”) 1.53e-7

Example (cont)

• Combining the probabilities, we can estimate:

P(“Eye halve a spelling chequer”) 6.12e-15P(“I have a spelling checker”) 1.53e-7

Yep, I have a spelling checker makes far more sense.

What else can we do?• Tell if a certain article is from the Washington Post or the

New York Times• Find the most informative sentence in a paragraph• Categorize texts into subjects (e.g. sports, economics,

literature, religion) • Tell that 2 news items are about the same event• Answer factual questions (When did Beethoven die?)• Divide sentences into meaningfull units[Pierre Vinken], [61 years old], [will join] [the board of

directors] [next Sunday]And much much more..

Boundaries disambiguation of coordinated conjunctions of NPs

And – what I’m interested in:

And – what I’m interested in

• I work on Ands.

• More specifically, I’m trying to figure out the boundaries of the things joined by Ands.

I ate green apples and juicy bananas for lunch

And – what I’m interested in

• Which of the following makes most sense?

I ate green (apples and juicy bananas) for lunch

I ate (green apples and juicy bananas) for lunch

I ate green (apples and juicy) bananas for lunch

I (ate green apples and juicy) bananas for lunch

…

And?

• This is very easy for people• This is very hard for computers

My main intuitions:

People are joining similar things

When they do so, they tend to use similar structure

Switching between the joined things is usually allowed

Thanks

Questions?