Markov Chains & Hidden Markov Models

Rainstorm 2020

M e r r i c k C a i & P h y l l i s Z h a n g

The Warren Buffett Problem

On his five-minute drive to the office, which he’s been making for the past five decades, Buffett stops by McDonald’s and orders one of three items: two sausage patties, a sausage, egg and cheese or a bacon, egg and cheese.

”$3.17 is a bacon, egg and cheese biscuit, but the market’s down this morning, so I’ll pass up the $3.17 and go with the $2.95.”

Introduction to Markov Chains

Let R and S be states. For now, we’ll call them State R and State S. From the arrows, we see that …

1. If we’re at State R, we can STAY at State R2. If we’re at State R, we can GO to state S

What else?

R S

Introduction to Markov Chains

These probabilities are called transition probabilities.

R S

1/2

2/3

1/21/3

Introduction to Markov Chains

We can also represent transition probabilities in a transition matrix.

R S

1/2

2/3

1/21/3

R S

R 1/3 2/3

S 1/2 1/2

Introduction to Markov Chains

R S

1/2

2/3

1/21/3

R S

R 1/3 2/3

S 1/2 1/2

- In the first row of this transition matrix, we can say that the probability of staying in state R is 1/3 and the probability of going from state R to state S is 2/3.

- Note that this probability adds to 1. From state R, we only have the two options; i.e. with probability 1 we will stay at R or go to S.

- What can we say about the second row?

Introduction to Markov Chains

R S

R 1/3 2/3

S 1/2 1/2

- Let’s give these states some meaning! - Suppose that on any day, the weather can be rainy or sunny. - An example following this transition matrix is…

- If today is rainy, then tomorrow will be sunny with probability 2/3.

Introduction to Markov Chains

R S

R 1/3 2/3

S 1/2 1/2

- What happens if tomorrow’s weather depends on both today and the day before today?

Introduction to Markov Chains

R S

R 1/3 2/3

S 1/2 1/2

A few brain teasers…

1. What is the probability, given that today is rainy, that tomorrow and the day after are both sunny?

2. What is the probability, given that yesterday and today are both rainy, that tomorrow and the day after are both sunny?

Introduction to Markov ChainsEXERCISE: Let’s draw our own Markov Chain.

Merrick is a happy traveler. He’s visiting Europe this summer, and he doesn’t really have a plan.He’s ready to let probability dictate where he will go!

Here are the locations he might visit on each day: - Venice- Milan- Pisa- Florence- London

Given the following statements, draw a Markov Chain representing Merrick’s state space and transition probabilities. Additionally, provide a transition matrix. You may use these states for simplicity: {V, M, P, F, L}

Introduction to Markov ChainsEXERCISE: Let’s draw our own Markov Chain.

State space: {V, M, P, F, L}

1. If Merrick is in Venice, he’ll stay in Venice with probability 1/3, go to Milan with probability 1/3, and go to Pisa with probability 1/3.

2. If Merrick is in Milan, he’ll stay in Milan with probability 1/2, go to Venice with probability 1/4, and go to Pisa with probability 1/4.

3. If Merrick is in Pisa, he’ll stay in Pisa with probability 3/4, go to Venice with probability 1/8, andgo to Milan with probability 1/8.

4. If Merrick is in Florence, he’ll stay in Florence with probability 1.5. If Merrick is in London, he’ll stay in London with probability 1/2 and go to Milan with

probability 1/2.

Introduction to Markov ChainsEXERCISE: Let’s draw our own Markov Chain.

State space: {V, M, P, F, L}

BrainTeasers:1. If Merrick starts his Europe trip in Milan and has an infinitely long vacation, which states are

reachable?

2. A state is transient if a chain starts at a state y and has a positive probability of never returning to y. Which states, if any, are transient?

3. If the probability Merrick begins his vacation at each of these cities is 1/5, what’s the probability he wastes his infinitely long vacation in the same city?

Hidden and Observed StatesLet’s return to our rainy sunny example.

R S

1/2

2/3

1/21/3

We’ll use the same setup, but now we’ll introduce…Merrick, a human being, and Oreo, a smol doggo.

New friends

Merrick

Oreo

Doesn’t see the weather, but sees Oreo

1. Is wet2. Is not wet

Merrick is currently being quarantined in his home. He can’t see the outside and has been pretty miserable lately. All he does is cook cabbage and feed Oreo.

Oreo, however, does go outside everyday! If it’s rainy outside, he’ll come back wet with probability 2/3. Merrick’s dad might provide him a poncho beforehand, keeping Oreo dry! If it’s sunny outside, he’ll come back wet with probability 1/4. Running through sprinkler systems is fun, even for us humans.

Hidden and Observed States

We’ll summarize our setup here.

This is our transition matrix.

R S

R 1/3 2/3

S 1/2 1/2

Hidden and Observed States

We’ll summarize our setup here.

This is our transition matrix.

R S

R 1/3 2/3

S 1/2 1/2

We can’t see the weather. Thus, the state of the weather is hidden to us. These hidden states are the basis of our Hidden Markov Model.

Hidden and Observed States

We’ll summarize our setup here.

This is our transition matrix.

R S

R 1/3 2/3

S 1/2 1/2

We can’t see the weather. Thus, the state of the weather is hidden to us. These hidden states are the basis of our Hidden Markov Model.

We CAN see Oreo! The two observed states are 1. Wet Oreo2. Dry Oreo

Hidden and Observed States

We’ll summarize our setup here.

This is our transition matrix.

R S

R 1/3 2/3

S 1/2 1/2

We can’t see the weather. Thus, the state of the weather is hidden to us. These hidden states are the basis of our Hidden Markov Model.

We CAN see Oreo! The two observed states are 1. Wet Oreo2. Dry Oreo

Now, emission probabilities are the probabilities that at each hidden state, we observe a certain state.

The emission probabilities in this case are …1. If it’s a rainy day, Oreo comes back wet with probability 2/3.2. If it’s a rainy day, Oreo comes back dry with probability ___.3. If it’s a sunny day, Oreo comes back wet with probability 1/4.4. If it’s a sunny day, Oreo comes back dry with probability ___.

Hidden and Observed States

Maximum LikelihoodQUESTION: If Oreo has come back Dry then Wet in the last two days, what was the most likely pattern of weather events? Assume that on the first day, Merrick checks the weather and sees that there is a 70% chance of precipitation.

Maximum LikelihoodQUESTION: If Oreo has come back Dry then Wet in the last two days, what was the most likely pattern of weather events? Assume that on the first day, Merrick checks the weather and sees that there is a 70% chance of precipitation.

4 Possibilities:

1. Rainy -> Sunny

2. Rainy -> Rainy

3. Sunny -> Sunny

4. Sunny -> Rainy

Maximum LikelihoodQUESTION: If Oreo has come back Dry then Wet in the last two days, what was the most likely pattern of weather events? Assume that on the first day, Merrick checks the weather and sees that there is a 70% chance of precipitation.

4 Possibilities:

1. Rainy -> Sunny

The probability of this occurring can be written as

P(1st day is Rainy) * P(Oreo is dry given a Rainy day) * P(2nd day is Sunny given 1st day is Rainy) * P(Oreo is wet given a Sunny day) = (0.7)(1/3)(2/3)(1/4) = 0.03889.

Maximum LikelihoodQUESTION: If Oreo has come back Dry then Wet in the last two days, what was the most likely pattern of weather events? Assume that on the first day, Merrick checks the weather and sees that there is a 70% chance of precipitation.

4 Possibilities:

2. Rainy -> Rainy

The probability of this occurring can be written as

P(1st day is Rainy) * P(Oreo is dry given a Rainy day) * P(2nd day is Rainy given 1st day is Rainy) * P(Oreo is wet given a Rainy day) = (0.7)(1/3)(1/3)(2/3) = 0.05185.

Maximum LikelihoodQUESTION: If Oreo has come back Dry then Wet in the last two days, what was the most likely pattern of weather events? Assume that on the first day, Merrick checks the weather and sees that there is a 70% chance of precipitation.

4 Possibilities:

3. Sunny -> Sunny

The probability of this occurring can be written as

P(1st day is Sunny) * P(Oreo is dry given a Sunny day) * P(2nd day is Sunny given 1st day is Sunny) * P(Oreo is wet given a Sunny day) = (0.3)(3/4)(1/2)(1/4) = 0.02813.

Maximum LikelihoodQUESTION: If Oreo has come back Dry then Wet in the last two days, what was the most likely pattern of weather events? Assume that on the first day, Merrick checks the weather and sees that there is a 70% chance of precipitation.

4 Possibilities:

4. Sunny -> Rainy

The probability of this occurring can be written as

P(1st day is Sunny) * P(Oreo is dry given a Sunny day) * P(2nd day is Rainy given 1st day is Sunny) * P(Oreo is wet given a Rainy day) = (0.3)(3/4)(1/2)(2/3) = 0.075.

Maximum LikelihoodQUESTION: If Oreo has come back Dry then Wet in the last two days, what was the most likely pattern of weather events? Assume that on the first day, Merrick checks the weather and sees that there is a 70% chance of precipitation.

4 Possibilities:

1. Rainy -> Sunny : 0.039

2. Rainy -> Rainy : 0.052

3. Sunny -> Sunny : 0.028

4. Sunny -> Rainy : 0.075

PREDICTION: The first day was Sunny and the second day was Rainy.

Maximum LikelihoodQUESTION: If Oreo has come back Wet, Wet, Dry, and Wet in the last 4 days, what was the most likely pattern of weather events? Assume that on the first day, Merrick checks the weather and sees that there is a 70% chance of precipitation.

Viterbi Algorithm QUESTION: If Oreo has come back Wet, Wet, Dry, and Wet in the last 4 days, what was the most likely pattern of weather events? Assume that on the first day, Merrick checks the weather and sees that there is a 70% chance of precipitation.

This problem is much more difficult to do than the previous one. Instead of having 4 cases, we now have 16cases. To solve larger and larger problems, people have developed many different types of algorithms to predict these hidden states.

While we won’t cover any of these algorithms in depth, we’ll give a quick introduction of the Viterbi Algorithm.

Viterbi Algorithm QUESTION: If Oreo has come back Wet, Wet, Dry, and Wet in the last 4 days, what was the most likely pattern of weather events? Assume that on the first day, Merrick checks the weather and sees that there is a 70% chance of precipitation.

WET WET DRY WET

Best path up to here

Best path up to here

Viterbi Algorithm QUESTION: If Oreo has come back Wet, Wet, Dry, and Wet in the last 4 days, what was the most likely pattern of weather events? Assume that on the first day, Merrick checks the weather and sees that there is a 70% chance of precipitation.

WET WET DRY WET

Best path up to here

Best path up to here

Viterbi Algorithm QUESTION: If Oreo has come back Wet, Wet, Dry, and Wet in the last 4 days, what was the most likely pattern of weather events? Assume that on the first day, Merrick checks the weather and sees that there is a 70% chance of precipitation.

WET WET DRY WET

Best path up to here

Best path up to here

Viterbi Algorithm QUESTION: If Oreo has come back Wet, Wet, Dry, and Wet in the last 4 days, what was the most likely pattern of weather events? Assume that on the first day, Merrick checks the weather and sees that there is a 70% chance of precipitation.

WET WET DRY WET

P(1st day is Sunny) * P(Oreo is wet given a Sunny day) = 0.3 * 0.25 = 0.075.

DAY 1 P(1st day is Rainy) * P(Oreo is wet given a Rainy day) = 0.7 * 0.67 = 0.469.

Viterbi Algorithm QUESTION: If Oreo has come back Wet, Wet, Dry, and Wet in the last 4 days, what was the most likely pattern of weather events? Assume that on the first day, Merrick checks the weather and sees that there is a 70% chance of precipitation.

WET WET DRY WET

0.075

DAY 2

0.469

Viterbi Algorithm QUESTION: If Oreo has come back Wet, Wet, Dry, and Wet in the last 4 days, what was the most likely pattern of weather events? Assume that on the first day, Merrick checks the weather and sees that there is a 70% chance of precipitation.

WET WET DRY WET

0.075

DAY 2

0.469

P(previous) * P(2nd day is Sunny given 1st is Sunny) * P(Oreo is wet given a Sunny day)= 0.075 * 0.5 * 0.25 = 0.009375.

P(previous) * P(2nd day is Sunny given 1st is Rainy) * P(Oreo is wet given a Sunny day)= 0.469 * 0.67 * 0.25 = 0.0786.

Viterbi Algorithm QUESTION: If Oreo has come back Wet, Wet, Dry, and Wet in the last 4 days, what was the most likely pattern of weather events? Assume that on the first day, Merrick checks the weather and sees that there is a 70% chance of precipitation.

WET WET DRY WET

0.075

DAY 2

0.469

0.0786

Viterbi Algorithm QUESTION: If Oreo has come back Wet, Wet, Dry, and Wet in the last 4 days, what was the most likely pattern of weather events? Assume that on the first day, Merrick checks the weather and sees that there is a 70% chance of precipitation.

WET WET DRY WET

0.075

DAY 2

0.469

Viterbi Algorithm QUESTION: If Oreo has come back Wet, Wet, Dry, and Wet in the last 4 days, what was the most likely pattern of weather events? Assume that on the first day, Merrick checks the weather and sees that there is a 70% chance of precipitation.

WET WET DRY WET

0.075

DAY 2

0.469

P(previous) * P(2nd day is Rainy given 1st is Sunny) * P(Oreo is wet given a Rainy day)= 0.075 * 0.5 * 0.67 = 0.0251.

P(previous) * P(2nd day is Rainy given 1st is Rainy) * P(Oreo is wet given a Rainy day)= 0.469 * 0.33 * 0.67 = 0.104.

0.104

Viterbi Algorithm QUESTION: If Oreo has come back Wet, Wet, Dry, and Wet in the last 4 days, what was the most likely pattern of weather events? Assume that on the first day, Merrick checks the weather and sees that there is a 70% chance of precipitation.

WET WET DRY WET

0.075

DAY 2

0.469 0.104

Viterbi Algorithm QUESTION: If Oreo has come back Wet, Wet, Dry, and Wet in the last 4 days, what was the most likely pattern of weather events? Assume that on the first day, Merrick checks the weather and sees that there is a 70% chance of precipitation.

WET WET DRY WET

0.075

0.469 0.104

0.0786

Viterbi Algorithm QUESTION: If Oreo has come back Wet, Wet, Dry, and Wet in the last 4 days, what was the most likely pattern of weather events? Assume that on the first day, Merrick checks the weather and sees that there is a 70% chance of precipitation.

WET WET DRY WET

0.469 0.104

Viterbi Algorithm QUESTION: If Oreo has come back following the table below in the last 14 days, what was the most likely pattern of weather events? Assume that on the first day, Merrick checks the weather and sees that there is a 70% chance of precipitation.

DRY DRY WET WET DRY WET WET

WET WET WET WET WET DRY WET

ANSWER:

SUNNY SUNNY RAINY RAINY SUNNY RAINY SUNNY

RAINY SUNNY RAINY SUNNY RAINY SUNNY RAINY

The Warren Buffett Problem

”$3.17 is a bacon, egg and cheese biscuit, but if the market’s down this morning, so I’ll pass up the $3.17 and go with the $2.95 (two sausage patties).”

The Warren Buffett ProblemLet’s use this as our transition matrix:

G B

G 1/2 1/2

B 3/4 1/4

Let’s use this as our emissions matrix:

Biscuit Patties

G 3/5 2/5

B 2/5 3/5

Let the beginning probability be 53.1% the market is up, 46.9% the market is down.

BISCUIT BISCUIT PATTIES BISCUIT PATTIES BISCUIT PATTIES

PATTIES BISCUIT BISCUIT BISCUIT PATTIES BISCUIT BISCUIT

LAST 14 PURCHASES BY WARREN BUFFETT:

The Warren Buffett Problem

BISCUIT BISCUIT PATTIES BISCUIT PATTIES BISCUIT PATTIES

PATTIES BISCUIT BISCUIT BISCUIT PATTIES BISCUIT BISCUIT

LAST 14 PURCHASES BY WARREN BUFFETT:

VITERBI PREDICTION

GOOD GOOD BAD GOOD BAD GOOD BAD

GOOD GOOD GOOD GOOD BAD GOOD GOOD

The Warren Buffett ProblemLet’s use this as our transition matrix:

G B

G 1/2 1/2

B 11/20 9/20

Let’s use this as our emissions matrix:

Biscuit Patties

G 3/4 1/4

B 1/3 2/3

Let the beginning probability be 53.1% the market is up, 46.9% the market is down.

BISCUIT BISCUIT PATTIES BISCUIT PATTIES BISCUIT PATTIES

PATTIES BISCUIT BISCUIT BISCUIT PATTIES BISCUIT BISCUIT

LAST 14 PURCHASES BY WARREN BUFFETT:

The Warren Buffett Problem

BISCUIT BISCUIT PATTIES BISCUIT PATTIES BISCUIT PATTIES

PATTIES BISCUIT BISCUIT BISCUIT PATTIES BISCUIT BISCUIT

LAST 14 PURCHASES BY WARREN BUFFETT:

VITERBI PREDICTION

GOOD GOOD BAD GOOD BAD GOOD BAD

BAD GOOD GOOD GOOD BAD GOOD GOOD

Markov Chains & Hidden Markov Models

Rainstorm 2020

M e r r i c k C a i & P h y l l i s Z h a n g