# Natural Language Processing

• View
33

0

Tags:

• #### sequence of random variables

Embed Size (px)

DESCRIPTION

Natural Language Processing. Spring 2007 V. “Juggy” Jagannathan. Course Book. Foundations of Statistical Natural Language Processing. By Christopher Manning & Hinrich Schutze. Chapter 9. Markov Models March 5, 2007. Markov models. Markov assumption - PowerPoint PPT Presentation

### Text of Natural Language Processing

• Natural Language ProcessingSpring 2007V. Juggy Jagannathan

• Foundations of Statistical Natural Language ProcessingByChristopher Manning & Hinrich SchutzeCourse Book

• Chapter 9Markov ModelsMarch 5, 2007

• Markov modelsMarkov assumptionSuppose X = (X1, , XT) is a sequence of random variables taking values in some finite set S = {s1,,sN}, Markov properties are:Limited HorizonP(Xt+1 = sk|X1,,Xt) = P(Xt+1 = sk|Xt)i.e. the t+1 value only depends on t valueTime invariant (stationary)Stochastic Transition matrix A:aij = P(Xt+1 = sj|Xt=si) where

• Markov model example

• Probability: {lem,ice-t} giventhe machine starts in CP?

0.3x0.7x0.1+0.3x0.3x0.7=0.021+0.063 = 0.084Hidden Markov Model Example

• Why use HMMs?Underlying events generating surface observable eventsEg. Predicting weather based on dampness of seaweedshttp://www.comp.leeds.ac.uk/roger/HiddenMarkovModels/html_dev/main.htmlLinear Interpolation in n-gram models:

• Look at Notes from David Meir Blei [UC Berkley]http://www-nlp.stanford.edu/fsnlp/hmm-chap/blei-hmm-ch9.pptSlides 1-13

• (Observed states)

• Forward Procedure

• Initialization:Induction:Total computation:Forward Procedure

• Initialization:Induction:Total computation:Backward Procedure

• Combining both forward and backward

• Finding the best state sequenceTo determine the state sequence that best explains observationsLet:Individually the most likely state is:This approach, however, does not correctly estimate the most likely state sequence.

• Finding the best state sequenceViterbi algorithmStore the most probable path that leads to a given nodeInitializationInductionStore Backtrace

• Parameter Estimation

• Parameter EstimationProbability of traversing an arc at time t given observation sequence O:

• Parameter Estimation Documents