Click here to load reader

Natural Language Processing

  • View

  • Download

Embed Size (px)


Natural Language Processing. Spring 2007 V. “Juggy” Jagannathan. Course Book. Foundations of Statistical Natural Language Processing. By Christopher Manning & Hinrich Schutze. Chapter 9. Markov Models March 5, 2007. Markov models. Markov assumption - PowerPoint PPT Presentation

Text of Natural Language Processing

  • Natural Language ProcessingSpring 2007V. Juggy Jagannathan

  • Foundations of Statistical Natural Language ProcessingByChristopher Manning & Hinrich SchutzeCourse Book

  • Chapter 9Markov ModelsMarch 5, 2007

  • Markov modelsMarkov assumptionSuppose X = (X1, , XT) is a sequence of random variables taking values in some finite set S = {s1,,sN}, Markov properties are:Limited HorizonP(Xt+1 = sk|X1,,Xt) = P(Xt+1 = sk|Xt)i.e. the t+1 value only depends on t valueTime invariant (stationary)Stochastic Transition matrix A:aij = P(Xt+1 = sj|Xt=si) where

  • Markov model example

  • Probability: {lem,ice-t} giventhe machine starts in CP?

    0.3x0.7x0.1+0.3x0.3x0.7=0.021+0.063 = 0.084Hidden Markov Model Example

  • Why use HMMs?Underlying events generating surface observable eventsEg. Predicting weather based on dampness of seaweeds Interpolation in n-gram models:

  • Look at Notes from David Meir Blei [UC Berkley] 1-13

  • (Observed states)

  • Forward Procedure

  • Initialization:Induction:Total computation:Forward Procedure

  • Initialization:Induction:Total computation:Backward Procedure

  • Combining both forward and backward

  • Finding the best state sequenceTo determine the state sequence that best explains observationsLet:Individually the most likely state is:This approach, however, does not correctly estimate the most likely state sequence.

  • Finding the best state sequenceViterbi algorithmStore the most probable path that leads to a given nodeInitializationInductionStore Backtrace

  • Parameter Estimation

  • Parameter EstimationProbability of traversing an arc at time t given observation sequence O:

  • Parameter Estimation