Natural Language Processing and Machine Learning for Discovery

MSU LawElectronic DiscoveryFal l 2012Week 9

NATURAL LANGUAGE PROCESSING AND MACHINE

LEARNING FOR DISCOVERY

Natural language processing Mathematical and linguistic concepts Models of representation Real-world application

Machine learning Common pre-processing and learning algorithms Real-world application

Communicate with software and service vendors!

GOALS

© Bommarito Consulting

Understand the BLACK BOX.

How do we characterize a black box?

BLACK BOX


3 English medium

Inputs Parameters Outputs

?

BLACK BOX


Secret: Most black boxes are very similar inside.

We’re going to learn to identify the common parts.

Defi nition: Dealing with real-world text in an automated, reproducible way.

Often referred to as NLP.

Used somewhat interchangeably with computational linguistics.

NATURAL LANGUAGE PROCESSING


Let’s start with some text.

“Hurricane Sandy grounded 3,200 flights scheduled for today and tomorrow, prompted New York to suspend subway and bus service and forced the evacuation of the New Jersey shore as it headed toward land with life-threatening wind and rain.

The system, which killed as many as 65 people in the Caribbean on its path north, may be capable of inflicting as much as $18 billion in damage when it barrels into New Jersey tomorrow and knock out power to millions for a week or more, according to forecasters and risk experts.”

(Bloomberg article on Sandy)



http://topics.bloomberg.com/new-york/


http://topics.bloomberg.com/new-jersey/

What kind of questions can we ask?

Basic What is the structure of the text?

Paragraphs Sentences Tokens/words

What are the words that appear in this text? Nouns

Subjects Direct objects

Verbs

Advanced What are the concepts that appear in this text? How does this text compare to other text?



Segmentation and Tokenization

“Hurricane Sandy grounded 3,200 fl ights scheduled for today and tomorrow, prompted New York to suspend subway and bus service and forced the evacuation of the New Jersey shore as it headed toward land with life-threatening wind and rain.

The system, which killed as many as 65 people in the Caribbean on its path north, may be capable of infl icting as much as $18 billion in damage when it barrels into New Jersey tomorrow and knock out power to millions for a week or more, according to forecasters and risk experts.”



• Segments Types• Paragraphs• Sentences• Tokens



Segmentation and Tokenization

But how does it work?

Paragraphs Two consecutive line breaks A hard line break followed by an indent

Sentences Period, except abbreviation, ellipsis within quotation, etc.

Tokens and Words Whitespace Punctuation

Remember what real-world text looks like – think text and email.



Segmentation and Tokenization“Hurricane Sandy grounded 3,200 fl ights scheduled for today and tomorrow, prompted New York to suspend subway and bus service and forced the evacuation of the New Jersey shore as it headed toward land with life-threatening wind and rain.

The system, which killed as many as 65 people in the Caribbean on its path north, may be capable of infl icting as much as $18 billion in damage when it barrels into New Jersey tomorrow and knock out power to millions for a week or more, according to forecasters and risk experts.”

Paragraphs: 2Sentences: 2Words: 561.

['Hurricane', 'Sandy', 'grounded', '3,200', 'fl ights', 'scheduled', 'for', 'today', 'and', 'tomorrow‘, …]





What kind of questions can we ask?

We now have an ordered list of tokens.

['Hurricane', 'Sandy', 'grounded', '3,200', 'fl ights', 'scheduled', 'for', 'today', 'and', 'tomorrow‘, …]

Does the word phrase “quote stuffi ng” occur in the text? How many times does “Sandy” occur? How often does “outage” occur after “power?” What percentage of tokens are numbers?



An Aside on Storage

Data: The word ‘the’ ten times and the word ‘a’ ten times.

Representation 1 - Ordered List: [‘the’, ‘a’, ‘the’, ‘a’, ‘the’, ‘a’, …]

Representation 2 – Term Frequency: [(‘the’, 10), (‘a’, 10)]



An Aside on Storage

Representation 1 - Ordered List: [‘the’, ‘a’, ‘the’, ‘a’, ‘the’, ‘a’, …]

Representation 2 - Frequency Map: [(‘the’, 10), (‘a’, 10)]

Tradeoffs Total space Ease of answering certain questions Information about context

Not all software make the same choice!



Stopwording, Stemming, Parsing, and Tagging

Stopwording Removing “fi ller” words like prepositions, auxiliary or infinitive

verbs, and conjunctions.

Stemming Matching declined nouns like dog/dogs or child/children. Matching conjugated verbs like run/ran.

Parsing Determining the “structure” of a sentence, typically as represented

by a grade school sentence diagram (requires grammar definition; we’ll skip).

Tagging Identifying the part of speech of each token in a sentence.



Stopwording Hurricane Sandy grounded 3,200 fl ights scheduled for today and tomorrow, prompted New York to suspend subway and bus service and forced the evacuation of the New Jersey shore as it headed toward land with life-threatening wind and rain.

The system, which killed as many as 65 people in the Caribbean on its path north, may be capable of infl icting as much as $18 billion in damage when it barrels into New Jersey tomorrow and knock out power to millions for a week or more, according to forecasters and risk experts.

Hurricane Sandy grounded 3,200 fl ights scheduled today tomorrow, prompted New York suspend subway bus service forced evacuation New Jersey shore headed toward land life-threatening wind rain.

System, killed many 65 people Caribbean path north, may capable infl icting much $18 billion damage barrels New Jersey tomorrow knock power millions week, according forecasters risk experts.





Stopwording + Stemming Hurricane Sandy grounded 3,200 fl ights scheduled for today and tomorrow, prompted New York to suspend subway and bus service and forced the evacuation of the New Jersey shore as it headed toward land with life-threatening wind and rain.


Hurrican Sandi ground 3,200 fl ight schedul today tomorrow, prompt New York suspend subway bu servic forc evacu New Jersey shore head toward land life-threaten wind rain.

System, kill mani 65 peopl Caribbean path north, may capabl infl ict much $18 billion damag barrel New Jersey tomorrow knock power million week, accord forecast risk expert.





Tagging Hurricane Sandy grounded 3,200 fl ights scheduled for today and tomorrow, prompted New York to suspend subway and bus service and forced the evacuation of the New Jersey shore as it headed toward land with life-threatening wind and rain.


[('Hurricane', 'NNP'), ('Sandy', 'NNP'), ('grounded', 'VBD'), ('3,200', 'CD'), ('fl ights', 'NNS'), ('scheduled', 'VBN'), ('for', 'IN'), ('today', 'NN'), ('and', 'CC'), ('tomorrow', 'NN'), …]





Back to the black box.



3 English medium

Inputs Parameters Outputs

Let’s say that we’re investigating Enron for accounting fraud related to its reserve reporting and transfers.

We want to look for any material that discusses reserves and profi ts in the same sentence. However, we want cases where these words are used as nouns; we’re not interested in dinner reservations.



Inputs Parameters Output

MemosResearchEmailsTextsTranscriptions

Stopword: NoStem: YesTag: YesSearch: …

MemosResearchEmailsTextsTranscriptions



In general, all document search and discovery software combines the elements discussed above.

Segment Tokenize Stopword Stem Parse Tag Store Search Retrieve



How do they diff er? Interface and ease-of-use De-duplication and versioning Supported languages Optical character recognition (OCR) File formats, e.g., Word, WordPerfect, PDF, HTML Ability to scale to large databases.

Definition: Automated classification and prediction on data.

Examples: Product recommenders, a la Amazon Computer vision – is it a cat? Sentiment analysis Topic classification Document clustering

At least two stages to machine learning: Training Classification

MACHINE LEARNING


Learning

Machine learning requires “learning” or “training.”

There are two types of training: Supervised Unsupervised

The goal of training is to determine a mapping from input features to a set of target classes.

MACHINE LEARNING


Learning

Imagine a student given a small l ist of organisms and descriptions. The student is tasked to assign the organisms into groups based on these descriptions. Where do the groups come from? Supervised: The teacher provides the answers. Unsupervised: The teacher provides nothing.

When the student is done with the task, the teacher checks the student’s responses and decides if the student has learned.

In our example, the teacher wil l typical ly provide the “canonical” domains and kingdoms of biology. However, most real-world problems domains are not so well-studied.

MACHINE LEARNING


Learning

What if the teacher gave the student some of the answers?

This is semi-supervised learning. Supervised: The teacher provides the answers.Semi-supervised: The teacher provides some answers.Unsupervised: The teacher provides nothing.

MACHINE LEARNING


Classification

The student has now learned to map from an organism’s description to a group. Now, the student is sent out into the field to use their knowledge to classify newly discovered organisms. They observe the organisms and document the features they learned to use. Then, they apply the learned rules to determine the class of organism.

MACHINE LEARNING


This is exactly how predictive coding works!

Organisms : DocumentsDescriptions : Natural language features or modelsSemi-supervised : Sample coding

The goal of predictive coding in discovery is to learn to classify documents based on natural language features, typically into relevant/irrelevant or privileged/unprivileged.

MACHINE LEARNING


Some Machine Learning Algorithms Supervised

Statistical models Bayesian, e.g., Naïve Bayes Classification Frequentist, e.g., Ordinary Least Squares.

Neural Networks (NN) Support Vector Machines (SVM) Random Forests (RF) Genetic Algorithms (GA)

Semi/unsupervised Neural Networks (NN) Clustering

K-means Hierarchical Radial Basis (RBF) Graph

MACHINE LEARNING


Notes on Algorithm Diversity

Not all algorithms return scores; some are binary. True, True, False 0.9, 0.7, 0.1

Not all algorithms support more than two classes. Cat, Dog, Mouse Cat, Not Cat

Not all algorithms scale similarly. 1M documents = 1 day 10M documents = {10 days, 100 days, 1000 days}

MACHINE LEARNING


Michael J Bommarito II CEO, Bommarito Consulting,

LLC Email:

[email protected] Web: http://bommaritollc.com/

THANKS!

You can get these slides on my blog – http://bommaritollc.com/blog/.


mailto:[email protected]

http://bommaritollc.com/

http://bommaritollc.com/blog/

http://bommaritollc.com/blog/

http://www.bommaritollc.com/

Books and Wiki Pages A Brief Survey of Text Mining. Hotho, Nurnberger, Paaß.

http://www.kde.cs.uni-kassel.de/hotho/pub/2005/hotho05TextMining.pdf Text Mining: Predictive Methods for Analyzing Unstructured Information. Weiss,

Indurkhya, Zhang, Damerau. http://www.amazon.com/Text-Mining-Predictive-Unstructured-Information/dp/0387954333

The Elements of Statistical Learning. http://www-stat.stanford.edu/~tibs/ElemStatLearn/

Wiki – Machine Learning. http://en.wikipedia.org/wiki/Machine_learning

Wiki – Machine Learning Algorithms. http://en.wikipedia.org/wiki/List_of_machine_learning_algorithms

Software Natural Language Toolkit (NLTK).

http://nltk.org/ Stanford NLP Group.

http://nlp.stanford.edu/software/ Weka.

http://www.cs.waikato.ac.nz/ml/weka/ R.

http://www.r-project.org/ SAS Predictive Analytics and Data Mining.

http://www.sas.com/technologies/analytics/datamining/index.html

REFERENCES

http://www.kde.cs.uni-kassel.de/hotho/pub/2005/hotho05TextMining.pdf



http://www.amazon.com/Text-Mining-Predictive-Unstructured-Information/dp/0387954333



http://www-stat.stanford.edu/~tibs/ElemStatLearn/

http://www-stat.stanford.edu/~tibs/ElemStatLearn/

http://en.wikipedia.org/wiki/Machine_learning



http://en.wikipedia.org/wiki/List_of_machine_learning_algorithms



http://nltk.org/

http://nltk.org/

http://nlp.stanford.edu/software/

http://nlp.stanford.edu/software/

http://www.cs.waikato.ac.nz/ml/weka/

http://www.cs.waikato.ac.nz/ml/weka/

http://www.r-project.org/

http://www.r-project.org/

http://www.sas.com/technologies/analytics/datamining/index.html

Technology

Natural Language Processing and Machine Learning for Discovery