1

A Quest for an Internet VideoQuality-of-Experience Metric

Athula Balachandran, Vyas Sekar,Aditya Akella, Srinivasan Seshan,

Ion Stoica, Hui Zhang

2

Internet Video is taking off

Improve Users’ Quality of Experience

3

Video Quality Metrics: The State of the Art

Objective Score (e.g., Peak Signal to Noise Ratio)

Subjective Scores(e.g., Mean Opinion

Score)

4

Problem 1: New Effects, New Metrics

PLAYERSTATES

EVENTS

Joining Playing Buffering Playing

Bufferfilled up

Bufferempty

Bufferfilled up

Switchbitrate

5

Problem 1: New Effects, New Metrics

PLAYERSTATES

EVENTS

Joining Playing Buffering Playing

Bufferfilled up

Bufferempty

Bufferfilled up

Switchbitrate

6

Problem 1: New Effects, New Metrics

PLAYERSTATES

EVENTS

Joining Playing Buffering Playing

Bufferfilled up

Bufferempty

Bufferfilled up

Switchbitrate

7

Problem 1: New Effects, New Metrics

PLAYERSTATES

EVENTS

Joining Playing Buffering Playing

Bufferfilled up

Bufferempty

Bufferfilled up

Switchbitrate

8

Problem 1: New Effects, New Metrics

PLAYERSTATES

EVENTS

Joining Playing Buffering Playing

Bufferfilled up

Bufferempty

Bufferfilled up

Switchbitrate

9

Problem 1: New Effects, New Metrics

PLAYERSTATES

EVENTS

Joining Playing Buffering Playing

Bufferfilled up

Bufferempty

Bufferfilled up

Switchbitrate

10

Problem 1: New Effects, New Metrics

PLAYERSTATES

EVENTS

Joining Playing Buffering Playing

Bufferfilled up

Bufferempty

Bufferfilled up

Switchbitrate

Join Time Buffering RatioRate of buffering

Rate of switchingAverage bitrate

11

Problem 2: Opinion Scores Engagement

Opinion Scores- Not representative of “in the wild” experience- Combinatorial explosion of parameters

Engagement as replacement for opinion score. (e.g., Play time, customer return rate)

12

Internet Video QoE

Objective ScoresPSNR

Subjective ScoresMOS

13

Internet Video QoE

Objective ScoresPSNR

Subjective ScoresMOS

Engagement(e.g., Fraction of video viewed)

14

Internet Video QoE

Objective ScoresPSNR

Join Time, Avg. bitrate, …?

Subjective ScoresMOS

Engagement(e.g., Fraction of video viewed)

15

Internet Video QoE

Objective ScoresPSNR

Join Time, Avg. bitrate, …?f(Join Time, Avg. bitrate, …)

Subjective ScoresMOS

Engagement(e.g., Fraction of video viewed)

16

Internet Video QoE

Objective ScoresPSNR

Join Time, Avg. bitrate, …?f(Join Time, Avg. bitrate, …)

Subjective ScoresMOS

Engagement(e.g., Fraction of video viewed)

17

Outline

• Need for a unified QoE

• What makes this hard?

• Our proposed approach

18

Challenge: Complex Engagement-to-metric Relationships

Enga

gem

ent

Quality Metric

[Dobrian et al. Sigcomm 2011] 19

Challenge: Complex Engagement-to-metric Relationships

Enga

gem

ent

Quality Metric

Non-monotonic

E

ngag

emen

t

Average bitrate

[Dobrian et al. Sigcomm 2011] 20

Challenge: Complex Engagement-to-metric Relationships

Enga

gem

ent

Quality Metric

Non-monotonic

E

ngag

emen

t

Average bitrate

En

gage

men

t

Rate of switching

Threshold

21

Challenge: Complex Metric Interdependencies

Join Time Bitrate

Rate of buffering

Rate of switching

Buffering Ratio

22

Challenge: Complex Metric Interdependencies

Join Time Bitrate

Rate of buffering

Rate of switching

Buffering Ratio

23

Challenge: Complex Metric Interdependencies

Join Time

Rate of buffering

Rate of switching

Buffering Ratio

Bitrate

24

Challenge: Complex Metric Interdependencies

Join Time Avg. bitrate

Rate of buffering

Rate of switching

Buffering Ratio

25

Need to learn these complex engagement-to-metric relationships and metric-to-metric dependencies

26

Casting as a Learning Problem

MACHINE LEARNING

Engagement Quality Metrics

QoE Model

Need to learn these complex engagement-to-metric relationships and metric-to-metric dependencies

27

Impact of the ML algorithm• Classify engagement into uniform classes• Accuracy = # of accurate predictions/ # of cases

ML algorithm must be expressive enough to handle the complex relationships and interdependencies

28

Challenge: Confounding Factors

Live and VOD sessions experience similar quality

29

Challenge: Confounding Factors

However, user viewing behavior is very different

30

Challenge: Confounding Factors

Devices User InterestConnectivity

Need systematic approach to identify and handle confounding factors

31

Domain-specific Refinement

MACHINE LEARNING

Engagement Quality Metrics

QoE Model

32

Domain-specific Refinement

MACHINE LEARNING

Engagement Quality Metrics

QoE Model

Confounding Factors

33

Improved prediction accuracy

Refined ML models can handle confounding factors

34

Concluding Remarks• Internet Video needs unified quantitative QoE

• What makes this hard?– Complex engagement-to-metric relationships– Complex metric-to-metric interdependencies– Confounding factors (e.g., genre, device)

• Promising start– Machine learning + domain-specific refinements

• Open Challenges– Coverage over confounding factors– System Design