Markov Chains in R

Divyansh SaxenaData Scientist @Trainline

“A stochastic model describing a sequence of possible events in which the probability of each event depends only on the state attained in the previous event.”

2

3

“A stochastic model describing a sequence of possible events in which the probability of each event depends only on the state attained in the previous event.”

Deterministic vs Stochastic Model

Markov Property

4

Deterministic Model Stochastic Model

In deterministic models, the output of the model is fully determined by the parameter values and the initial conditions.

Eg - Calculation to determine the return on a 5-year investment with an annual interest rate of 7%, compounded monthly.

The model is just the equation below

F = P(1 + r/m)Ym

Stochastic models possess some inherent randomness. The same set of parameter values and initial conditions will lead to an ensemble of different outputs.

Eg - A random walk process

Path A

Path B

Path C

55

“A stochastic model describing a sequence of possible events in which the probability of each event depends only on the state attained in the previous event.”

Deterministic vs Stochastic Model

Markov Property

6

Markov Property

In probability theory and statistics, the term Markov property refers to the memoryless property of a stochastic process.

A stochastic process has the Markov Property if the conditional probability distribution of the future states of the process (conditional on both past and present states) depends only upon the present state, not on the sequence of events that preceded it.

P(Xn = xn | Xn-1 = xn-1, . . . , X0 = x0) = P(Xn = xn | Xn-1 = xn-1)

Marketing Attribution

8

“In marketing, attribution is the identification of a set of user actions ("events" or "touchpoints") that contribute in some manner to a desired outcome, and then the

assignment of a value to each of these events.”

Display PPC SEO Conversion No ConversionDisplay PPC SEO No Conversion

9

What is the contribution of each unique ad type towards conversion?

33%

33%

33%50% 50% 100%OR OR

SEOPPCDisplay

10

Types of Attribution

Single Source Attribution

FractionalAttribution

Algorithmic / ProbabilisticAttribution

Last Click

First Click

Last Touch

Markov Chains

SurvivalModel

Recurrent Networks

11

Let’s assume 3 customer journeys

Display PPC SEO Conversion No ConversionDisplay PPC SEO No Conversion

1 2 3

Last Click Method

12

Display PPC SEO Conversion

100%

0% 0%

Markov Chain Method

12

1 - Customer Journey 2 - Transformation 3 - Splitting for pairs

Display > PPC > SEO > Conversion

Display

PPC > SEO

(start) > Display > PPC > SEO > (conversion)(start) > Display, Display > PPC, PPC > SEO, SEO > (conversion)

(start) > Display > (null)

(start) > PPC > SEO > (null)

(start) > Display, Display > (null)

(start) > PPC, PPC > SEO, SEO > (null)

Transition Matrix

from to probability total probability

(start) Display ⅓

(start) Display ⅓2/3

(start) PPC ⅓ 1/3

Display PPC ½

Display (null) ½

1/2

1/2

PPC SEO ½

PPC SEO ½ 1

SEO (conversion) ½

SEO (null) ½

1/2

1/2

13

Markov Chain

(start)

Display

PPC

(null)

(conversion) SEO

66.7%

33.3%

100% 50%

50%

50%

50%

14

15

What is the total conversion probability?

16

(start)

Display

PPC

(null)

(conversion) SEO

66.7%

33.3%

100% 50%

50%

50%

50%

Total Conversion Probability = 66.7%

17

(start)

Display

PPC

(null)

(conversion) SEO

66.7%

33.3%

100% 50%

50%

50%

50%

Total Conversion Probability = 66.7% x 50%

18

(start)

Display

PPC

(null)

(conversion) SEO

66.7%

33.3%

100% 50%

50%

50%

50%

Total Conversion Probability = 66.7% x 50% x 100%

19

(start)

Display

PPC

(null)

(conversion) SEO

66.7%

33.3%

100% 50%

50%

50%

50%

Total Conversion Probability = 66.7% x 50% x 100% x 50%

20

Total Conversion Probability = (66.7% x 50% x 100% x 50%) + (33.3%)

(start)

Display

PPC

(null)

(conversion) SEO

66.7%

33.3%

100% 50%

50%

50%

50%

21

Total Conversion Probability = (66.7% x 50% x 100% x 50%) + (33.3% x 100%)

(start)

Display

PPC

(null)

(conversion) SEO

66.7%

33.3%

100% 50%

50%

50%

50%

22

Total Conversion Probability = (66.7% x 50% x 100% x 50%) + (33.3% x 100% x 50%)

= (16.7% + 16.7%)

= 33.3%

(start)

Display

PPC

(null)

(conversion) SEO

66.7%

33.3%

100% 50%

50%

50%

50%

23

What is the contribution of each unique ad type towards conversion?

33%

33%

33%50% 50% 100%OR OR

SEOPPCDisplay

Removal Effects

25

Removal Effects

If we were to figure out what is the contribution of Display in our customer’s journey from start to end conversion, we will use the principle of removal effect.

Removal effect principle says that if we want to find the contribution of each channel in the customer journey, we can do so by removing each channel and see how many conversions are happening without that channel being in place.

26

Removal Effect of Display

(start)

Display

PPC

(null)

(conversion) SEO

66.7%

33.3%

100% 50%

50%

50%

50%

27

Removal Effect of Display

(start)

PPC

(null)

(conversion) SEO

33.3%

100% 50%

50%

28

Removal Effect of Display

(start)

PPC

(null)

(conversion) SEO

33.3%

100% 50%

50%

Conversion probability after removing Display = (33.3% x 100% x 50%)

= 16.7%

29

Removal Effect of Display

(start)

PPC

(null)

(conversion) SEO

33.3%

100% 50%

50%

Conversion probability after removing Display = 16.7%

Removal Effect of Display = (33.3% - 16.7%) / 33.3%

= 50%

30

Removal Effect of PPC

(start)

Display

PPC

(null)

(conversion) SEO

66.7%

33.3%

100% 50%

50%

50%

50%

31

Removal Effect of PPC

(start)

Display (null)

(conversion) SEO

66.7%

50%

50%

50%

35

Removal Effect of PPC

(start)

Display (null)

(conversion) SEO

66.7%

50%

50%

50%

Conversion probability after removing PPC = 0%

Removal Effect of PPC = (33.3% - 0%) / 33.3%

= 100%

36

Removal Effect of SEO

(start)

Display

PPC

(null)

(conversion) SEO

66.7%

33.3%

100% 50%

50%

50%

50%

37

Removal Effect of SEO

(start)

Display

PPC

(null)

(conversion)

66.7%

33.3%

50%

50%

38

Removal Effect of SEO

(start)

Display

PPC

(null)

(conversion)

66.7%

33.3%

50%

50%

Conversion probability after removing SEO = 0%

Removal Effect of SEO = (33.3% - 0%) / 33.3%

= 100%

Removal Effect of each ad type / channel

50% 100% 100%

Display PPC SEO

39

Contribution of Display

50% 100% 100%

Display PPC SEO

40

50%

Display

Contribution of Display

50% 100% 100%

Display PPC SEO

41

50%

Display 20%

Contribution of each ad type / channel using Markov Chain Method

20% 40% 40%

Display PPC SEO

42

Results

44

Display PPC SEO Conversion No ConversionDisplay PPC SEO No Conversion

Last Click Method Markov Chain Method

0% 0% 100%

Display PPC SEO

20% 40% 40%

Display PPC SEO

Markov Chains in R

46

Package in CRAN - ChannelAttribution

Two functions - Markov Model and Heuristic Model

Last TouchFirst Touch

47

A customer converted after interacting with 4 channels / ad types.

48

channel_9 > channel_6 > channel_0 > channel_1

49

Markov Model in R

50

Results TransitionMatrix

Removal Effects

51

Results TransitionMatrix

Removal Effects

52

Results TransitionMatrix

Removal Effects

53

Results TransitionMatrix

Removal Effects

54

Comparison of Markov Model result result with Heuristic Model result

Conclusion

56

Simple&

Easy to Interpret

57

Simple&

Easy to InterpretPowerfullyPredictive

58

Simple&

Easy to InterpretPowerfullyPredictive

Data - Driven