Conditional Random Fields

By:Vidya Venkiteswaran

AGENDA

What is CRFsuite?CRFsuite is an implementation of Conditional Random Fields (CRFs)

CRFs(conditional random fields) is a popular probabilistic method for structured prediction.

Common funda behind many applications is the ability to predict multiple variables that depend on each other

What does it do?Imagine you have a sequence of snapshots from a day in your life, and you want to label each image with the activity it represents (eating, sleeping, driving, etc.). How can you do this?

One way is to ignore the sequential nature of the snapshots, and build a per-image classifier. For example, given a month’s worth of labeled snapshots, you might learn that dark images taken at 6am tend to be about sleeping, images with lots of bright colors tend to be about dancing, images of cars are about driving, and so on.

By ignoring this sequential aspect, however, you lose a lot of information. For example, what happens if you see a close-up picture of a mouth – is it about singing or eating? If you know that the previous image is a picture of you eating or cooking, then it’s more likely this picture is about eating; if, however, the previous image contains you singing or dancing, then this one probably shows you singing as well.

Thus, to increase the accuracy of our labeler, we should incorporate the labels of nearby photos, and this is precisely what a conditional random field does.

Why do we need it?In many applications, we want the ability to predict multiple variables that depend on each other. For example, the performance of a sports team depends on the health of each member of the team. The health of each member might get affected by the travel schedule of the team. The results of the game might affect the morale of the team. The morale, in turn, might affect the health. As you can see, there are multiple variables that depend on each other intricately. Conditional Random Fields (CRFs) are very useful in modeling these problems.

Examples & Application

• Segmenting genes in a strand of DNA• Extracting syntax from natural-language text• Classifying regions of an image• Estimating the score in a strategic game

Importance of ContextProblem Statement:

Given a sentence, we have to label the words as Nouns / Verbs / Others.

For example:X = I play CricketY = [Noun, Verb, Noun]

X = Fish don’t sleepY = [Noun, Other, Verb]

X = I like to fishY = [Noun, Other, Other, Verb]

Notice the labelling of the word “Fish” here.It can be both Noun or Verb depending upon the context.Hence, relative ordering of tokens in sequence is very important for labeling.

Alternatives to CRFSuite

• Graphical Models• Hidden Markov Models (HMMs)

Generative vs Discriminative Models

Generative Models Discriminative Models

Joint distribution: P(x,y) Conditional distribution: P(y|x)

Use Baye’s Theorem to predict P(y|x) Can directly predict P(y|x)

ExamplesNaive Multiclass

Hidden Markov Models

ExamplesLogistic Regression

Conditional Random Fields

Linear functions Exponential functions

Easy to train but less accurate Hard to train but more accurate

Relationship with other models

ImplementationGiven a sequence X = [x1, x2, …, xN],We have to predict the corresponding labels Y = [y1, y2, …, yN]

Step 1: Creation of feature vectors from the given sequence F = [f1, f2, …, fN]

Step 2: Calculate the probability of each possible label yi given a feature vector fi

Step 3: The yi with maximum probability for given feature fi is the label of token xi

Label of xi = argmax( P( yi | fi ) )

References

• http://homepages.inf.ed.ac.uk/csutton/publications/crftut-fnt.pdf

• http://www.yisongyue.com/courses/cs155/lectures/Lecture_09.pdf

• http://courses.washington.edu/ling572/winter2016/teaching_slides/class13_crf.pdf

• http://homepages.inf.ed.ac.uk/csutton/publications/crftut-fnt.pdf