viterbi algorithm - Emory Universitychoi/courses/cs325/slides/viterbi_algorithm.… · Viterbi Algorithm Data Structures and Algorithms ... 'OQT[oU RTKOCT[ EQNQTU CTG 'OQT[ DNWG 2/5

Emory University Logo Guidelines

#P�KPUVKVWVKQP�CU�NCTIG�CPF�XCTKGF�CU�'OQT[�TGSWKTGU�C�EQPUKUVGPV�XKUWCN�KFGPVKV[�VJCV�WPKƂGU�KVU�XCTKQWU�CHƂNKCVGU��'OQT[oU�EWTTGPV�UVCPFCTFU��YJKEJ�JCXG�DGGP�KP�WUG�UKPEG��TGKPHQTEG�VJG�WPKSWG�EJCTCEVGT�CPF�SWCNKV[�QH�GCEJ�CECFGOKE�CPF�CFOKPKUVTCVKXG�WPKV��YJKNG�UKOWNVCPGQWUN[�OCMKPI�KV�ENGCT�VJCV�'OQT[�UVCPFU�DGJKPF�GCEJ�QH�VJGO� +P�CFFKVKQP�VQ�VJG�OCKP�7PKXGTUKV[�ITCRJKE�KFGPVKƂGTU��OQUV�UEJQQNU�CPF�OCLQT�WPKVU�JCXG�VJGKT�QYP�EQORNGOGPVCT[�UGV�QH�KFGPVKV[�ITCRJKEU�HQT�RTKPV�CPF�YGD��YJKEJ�YGTG�FGXGNQRGF�KP�ECTGHWN�EQPUWNVC-VKQP�YKVJ�FGCPU�CPF�WPKV�JGCFU��&QYPNQCFCDNG�NQIQU�CPF�YQTFOCTMU�HQT�'OQT[�7PKXGTUKV[��VJG�UEJQQNU��CPF�OCLQT�WPKVU�ECP�DG�HQWPF�QP�VJG�YGD�CV�JVVR��KFGPVKV[�GOQT[�GFW��RTKPV�ITCRJKE�UVCPFCTFU��CPF�JVVR��YGDIWKFG�GOQT[�GFW��YGD�ITCRJKE�UVCPFCTFU��6JG�'/14;�YQTFOCTM�KU�C�HGFGTCNN[�TGIKUVGTGF�VTCFGOCTM��#�UCPEVKQPGF�KFGPVKƂGT�QH�VJG�7PKXGTUKV[tC�UEJQQN�QT�OCLQT�WPKV�NQIQ�VJCV�KPENWFGU�VJG�UJKGNF�U[ODQN�CPF�VJG�YQTFOCTM�'/14;tUJQWNF�CRRGCT�QP�GCEJ�RWDNKECVKQP��+H�[QW�YKUJ�VQ�JCXG�CP�KFGPVKƂGT�WPKV�UKIPCVWTG��ETGCVGF�URGEKƂECNN[�HQT�[QWT�RTQITCO�QT�FGRCTVOGPV��RNGCUG�EQPVCEV�VJG�1HƂEG�QH�$TCPF�/CPCIGOGPV�CV��QT�UVCPKU�MQFOCP"GOQT[�GFW��

6JG�V[RGHCEG�)QWF[�KU�TGUGTXGF�HQT�VJG�'OQT[�VTCFGOCTMU�CPF�PGXGT�UJQWNF�DG�WUGF�KP�VGZV�QT�FKURNC[�EQR[��;QW�PGXGT�UJQWNF�CVVGORV�VQ�TGPFGT�VJG�'OQT[�NQIQ�D[�V[RKPI�VJG�NGVVGTU�KP�C�YQTF�RTQEGUUKPI�QT�RCIG�NC[QWV�RTQITCO��0QPG�QH�VJG�NQIQU�KU�C�V[RGF�YQTF�DWV�TCVJGT�KU�URGEKƂECNN[�FGUKIPGF�XGEVQT�CTV�

'OQT[�VKGT�QPG�NQIQU�UJQWNF�TGRTQFWEG�QPN[�KP�'OQT[�DNWG�2/5��DNCEM��QT�YJKVG��6JG�UJKGNF�JGKIJV�UJQWNF�TGRTQFWEG�CV��q�QT�NCTIGT��#�IGPGTCN�TWNG�HQT�URCEKPI�CTQWPF�CP�'OQT[�NQIQ�KU�VQ�KPVGITCVG�CP�QDXKQWU�XKUWCN�UGRCTCVKQPtPQ�FGUKIP�GNGOGPV�QT�VGZV�UJQWNF�DG�PGUVGF�YKVJ�'OQT[�NQIQU�

'OQT[oU�RTKOCT[�EQNQTU�CTG�'OQT[�DNWG�2/5��CPF�[GNNQY�2/5��'OQT[�7PKXGTUKV[�YQTFOCTMU� ECP�DG�TGRTQFWEGF�KP�'OQT[�DNWG�2/5��DNCEM��QT�YJKVG�QP�CP�'OQT[�DNWG�QT�FCTM�DCEMITQWPF��

YYY�KFGPVKV[�GOQT[�GFWJVVR��YGDIWKFG�GOQT[�GFW

3/8” minimun reproductionheight of shield

Keep a space around the logo equal to the height and width of the “M” in Emory

PMS 280 PMS 131 PMS 130 coated uncoated

Web colors are: Emory blue 002878 gold (dark) d28e00 gold (light) d2b000

Viterbi Algorithm

Data Structures and AlgorithmsEmory University

Jinho D. Choi











Viterbi Algorithm

2

John did not











Viterbi Algorithm

2

John did not

S











Viterbi Algorithm

2

John did not

n

v

a

p

r

S











Viterbi Algorithm

2

John did not

n

v

a

p

r

S

1/6⋅1ε⋅ε

ε⋅ε

ε⋅ε

ε⋅ε











Viterbi Algorithm

2

John did not

n

v

a

p

r

S

1/6⋅1ε⋅ε

ε⋅ε

ε⋅ε

ε⋅ε

1/6

ε2

ε2

ε2

ε2











Viterbi Algorithm

2

John did not

n

v

a

p

r

S

1/6⋅1ε⋅ε

ε⋅ε

ε⋅ε

ε⋅ε

1/6

ε2

ε2

ε2

ε2

n

v

a

p

r











Viterbi Algorithm

2

John did not

n

v

a

p

r

S

1/6⋅1ε⋅ε

ε⋅ε

ε⋅ε

ε⋅ε

1/6

ε2

ε2

ε2

ε2

n

v

a

p

r











Viterbi Algorithm

2

John did not

n

v

a

p

r

S

1/6⋅1ε⋅ε

ε⋅ε

ε⋅ε

ε⋅ε

1/6

ε2

ε2

ε2

ε2

n

v

a

p

r

ε⋅1/6

ε⋅2/6











Viterbi Algorithm

2

John did not

n

v

a

p

r

S

1/6⋅1ε⋅ε

ε⋅ε

ε⋅ε

ε⋅ε

1/6

ε2

ε2

ε2

ε2

n

v

a

p

r

ε⋅1/6

ε⋅2/6











Viterbi Algorithm

2

John did not

n

v

a

p

r

S

1/6⋅1ε⋅ε

ε⋅ε

ε⋅ε

ε⋅ε

1/6

ε2

ε2

ε2

ε2

n

v

a

p

r

1/6⋅ε⋅1/6

1/6⋅ε⋅2/6

ε⋅1/6

ε⋅2/6











Viterbi Algorithm

2

John did not

n

v

a

p

r

S

1/6⋅1ε⋅ε

ε⋅ε

ε⋅ε

ε⋅ε

1/6

ε2

ε2

ε2

ε2

n

v

a

p

r

1/6⋅ε⋅1/6

1/6⋅ε⋅2/6

ε⋅1/6

ε⋅2/6

n

v

a

p

r











Viterbi Algorithm

2

John did not

n

v

a

p

r

S

1/6⋅1ε⋅ε

ε⋅ε

ε⋅ε

ε⋅ε

1/6

ε2

ε2

ε2

ε2

n

v

a

p

r

1/6⋅ε⋅1/6

1/6⋅ε⋅2/6

ε⋅1/6

ε⋅2/6

n

v

a

p

r











Viterbi Algorithm

2

John did not

n

v

a

p

r

S

1/6⋅1ε⋅ε

ε⋅ε

ε⋅ε

ε⋅ε

1/6

ε2

ε2

ε2

ε2

n

v

a

p

r

1/6⋅ε⋅1/6

1/6⋅ε⋅2/6

ε⋅1/6

ε⋅2/6

n

v

a

p

r

1/6⋅1/3

ε⋅1/3

ε⋅1/3











Viterbi Algorithm

2

John did not

n

v

a

p

r

S

1/6⋅1ε⋅ε

ε⋅ε

ε⋅ε

ε⋅ε

1/6

ε2

ε2

ε2

ε2

n

v

a

p

r

1/6⋅ε⋅1/6

1/6⋅ε⋅2/6

ε⋅1/6

ε⋅2/6

n

v

a

p

r

1/6⋅1/3

ε⋅1/3

ε⋅1/3











Viterbi Algorithm

2

John did not

n

v

a

p

r

S

1/6⋅1ε⋅ε

ε⋅ε

ε⋅ε

ε⋅ε

1/6

ε2

ε2

ε2

ε2

n

v

a

p

r

1/6⋅ε⋅1/6

1/6⋅ε⋅2/6

ε⋅1/6

ε⋅2/6

n

v

a

p

r

1/6⋅1/3

ε⋅1/3

ε⋅1/3

1/6⋅1/2

ε⋅1/2











Viterbi Algorithm

2

John did not

n

v

a

p

r

S

1/6⋅1ε⋅ε

ε⋅ε

ε⋅ε

ε⋅ε

1/6

ε2

ε2

ε2

ε2

n

v

a

p

r

1/6⋅ε⋅1/6

1/6⋅ε⋅2/6

ε⋅1/6

ε⋅2/6

n

v

a

p

r

1/6⋅1/3

ε⋅1/3

ε⋅1/3

1/6⋅1/2

ε⋅1/2

1/6⋅ε⋅1/6⋅1/6⋅1/3

1/6⋅ε⋅2/6⋅1/6⋅1/2











Viterbi Algorithm

2

John did not

n

v

a

p

r

S E

1/6⋅1ε⋅ε

ε⋅ε

ε⋅ε

ε⋅ε

1/6

ε2

ε2

ε2

ε2

n

v

a

p

r

1/6⋅ε⋅1/6

1/6⋅ε⋅2/6

ε⋅1/6

ε⋅2/6

n

v

a

p

r

1/6⋅1/3

ε⋅1/3

ε⋅1/3

1/6⋅1/2

ε⋅1/2

1/6⋅ε⋅1/6⋅1/6⋅1/3

1/6⋅ε⋅2/6⋅1/6⋅1/2











Viterbi Algorithm

2

John did not

n

v

a

p

r

S E

1/6⋅1ε⋅ε

ε⋅ε

ε⋅ε

ε⋅ε

1/6

ε2

ε2

ε2

ε2

n

v

a

p

r

1/6⋅ε⋅1/6

1/6⋅ε⋅2/6

ε⋅1/6

ε⋅2/6

n

v

a

p

r

1/6⋅1/3

ε⋅1/3

ε⋅1/3

1/6⋅1/2

ε⋅1/2

1/6⋅ε⋅1/6⋅1/6⋅1/3

1/6⋅ε⋅2/6⋅1/6⋅1/2











Viterbi Algorithm

2

John did not

n

v

a

p

r

S E

1/6⋅1ε⋅ε

ε⋅ε

ε⋅ε

ε⋅ε

1/6

ε2

ε2

ε2

ε2

n

v

a

p

r

1/6⋅ε⋅1/6

1/6⋅ε⋅2/6

ε⋅1/6

ε⋅2/6

n

v

a

p

r

1/6⋅1/3

ε⋅1/3

ε⋅1/3

1/6⋅1/2

ε⋅1/2

1/6⋅ε⋅1/6⋅1/6⋅1/3

1/6⋅ε⋅2/6⋅1/6⋅1/2











Viterbi Algorithm

3











Viterbi Algorithm• Dynamic programming

3












- Save the best path for each token.

3













- Back-track the best sequence.

3














• Proof of correctness

3















- By induction.

3















- By induction.

- Base: ?

3















- By induction.

- Base: ?

- Induction: ?

3















- By induction.

- Base: ?

- Induction: ?

3

ˆt1 = argmax

t1P (t1|t0) · P (w1|t1)















- By induction.

- Base: ?

- Induction: ?

3

ˆt1 = argmax

t1P (t1|t0) · P (w1|t1)

ˆtk1 = (

ˆtk�1, argmax

tkP (tk|tk�1) · P (wk|tk))











HMM Tagger

4

public class HMMTagger extends AbstractTagger{

static public String START_STATE = "START_STATE";static public String END_STATE = "END_STATE";private HiddenMarkov hm_model;

public HMMTagger(double epsilon){

hm_model = new HiddenMarkov(epsilon);}

}











HMM Tagger

4





}











HMM Tagger

4





}

cs325/classifier/HiddenMarkov.javacs325/utils/IntDoublePair.java











HMM Tagger

5

public void addSentence(List<String> words, List<String> tags){

String currenetState, previousState = START_STATE;int i, size = words.size();

for (i=0; i<size; i++){

currenetState = tags.get(i);hm_model.addTransition (previousState, currenetState);hm_model.addObservation(currenetState, words.get(i));previousState = currenetState;

}

hm_model.addTransition(previousState, END_STATE);}

@Overridepublic void train(){

hm_model.train();}











HMM Tagger

5





}



hm_model.train();}











HMM Tagger

5





}



hm_model.train();}











HMM Tagger

5





}



hm_model.train();}











Initialization

6

Dynamic Table

John did not like playing with Tomnoun 1⋅1/6verb ε2

aux ε2

prep ε2

adv ε2

Backpointer Table

John did not like playing with Tomnoun 0verb 0aux 0prep 0adv 0











Initialization

6

Dynamic Table


aux ε2

prep ε2

adv ε2

Backpointer Table


P (t1|tS) · P (w1|t1)











Initialization

6

Dynamic Table


aux ε2

prep ε2

adv ε2

Backpointer Table


P (t1|tS) · P (w1|t1) Intermediate probabilities











Initialization

6

Dynamic Table


aux ε2

prep ε2

adv ε2

Backpointer Table













Initialization

6

Dynamic Table


aux ε2

prep ε2

adv ε2

Backpointer Table



Best previous state











Initialization

7

public List<TagList> decode(List<String> observations){

final int N = hm_model.getLabels().size(); final int T = observations.size();

double[][] viterbi = new double[N][T];int[][] backpointer = new int[N][T];IntDoublePair max;int s, t = 0;

// initializationfor (s=0; s<N; s++){

viterbi[s][0] = hm_model.getOverallLikelihood(START_STATE, hm_model.getLabel(s), observations.get(t));

backpointer[s][0] = 0;}

vs,0 = P (ts|tSTART ) · P (w1|ws)bs,0 = 0











Initialization

7

public List<TagList> decode(List<String> observations){

final int N = hm_model.getLabels().size(); final int T = observations.size();

double[][] viterbi = new double[N][T];int[][] backpointer = new int[N][T];IntDoublePair max;int s, t = 0;

// initializationfor (s=0; s<N; s++){

viterbi[s][0] = hm_model.getOverallLikelihood(START_STATE, hm_model.getLabel(s), observations.get(t));

backpointer[s][0] = 0;}











Recursion

8

Dynamic Table


aux ε2

prep ε2

adv ε2

Backpointer Table












Recursion

8

Dynamic Table


aux ε2

prep ε2

adv ε2

Backpointer Table












Recursion

9

// recursionfor (t=1; t<T; t++){

for (s=0; s<N; s++){

max = max(viterbi, t, hm_model.getLabel(s), observations.get(t), N);viterbi[s][t] = max.getDouble();backpointer[s][t] = max.getInt();

}}

private IntDoublePair max(double[][] viterbi, int t, String currentState, String observation, final int N){

}











Recursion

9


for (s=0; s<N; s++){


}}


}

bs,t = argmax

s(vs0,t�1 · P (s|s0) · P (wt))

vs,t = max(vs0,t�1 · P (s|s0) · P (wt))











Recursion

9


for (s=0; s<N; s++){


}}


}

bs,t = argmax

s(vs0,t�1 · P (s|s0) · P (wt))

vs,t = max(vs0,t�1 · P (s|s0) · P (wt))











Recursion

10


int s = 0;double d = viterbi[s][t-1] *

hm_model.getOverallLikelihood(hm_model.getLabel(s), currentState, observation);IntDoublePair max = new IntDoublePair(s, d);

for (s=1; s<N; s++){ d = viterbi[s][t-1] *

hm_model.getOverallLikelihood(hm_model.getLabel(s), currentState, observation); if (d > max.getDouble()) max.set(s, d);}

return max;}











Recursion

10






return max;}











Recursion

10






return max;}











Termination

11

Dynamic Table


aux ε2

prep ε2

adv ε2

Backpointer Table












Termination

11

Dynamic Table


aux ε2

prep ε2

adv ε2

Backpointer Table












Termination

11

Dynamic Table


aux ε2

prep ε2

adv ε2

Backpointer Table












Termination

11

Dynamic Table


aux ε2

prep ε2

adv ε2

Backpointer Table












Termination

11

Dynamic Table


aux ε2

prep ε2

adv ε2

Backpointer Table












Termination

11

Dynamic Table


aux ε2

prep ε2

adv ε2

Backpointer Table












Termination

11

Dynamic Table


aux ε2

prep ε2

adv ε2

Backpointer Table












Termination

11

Dynamic Table


aux ε2

prep ε2

adv ε2

Backpointer Table












Termination

11

Dynamic Table


aux ε2

prep ε2

adv ε2

Backpointer Table












Termination

12

max = max(viterbi, t, END_STATE, null, N);

private TagList getTags(int[][] backpointer, int max, final int N, final int T){

List<String> tags = new ArrayList<>();int t;

tags.add(hm_model.getLabel(max));

for (t=T-1; t>0; t--){

max = backpointer[max][t];tags.add(hm_model.getLabel(max));

}

Collections.reverse(tags);return new TagList(tags);

}











Termination

12





for (t=T-1; t>0; t--){


}


}











Termination

12





for (t=T-1; t>0; t--){


}


}











Termination

12





for (t=T-1; t>0; t--){


}


}

Documents

viterbi algorithm - Emory Universitychoi/courses/cs325/slides/viterbi_algorithm.… · Viterbi Algorithm Data Structures and Algorithms ... 'OQT[oU RTKOCT[ EQNQTU CTG 'OQT[ DNWG 2/5