UNDERSTANDING THE IMPACT OF VIDEO QUALITY ON USER ENGAGEMENT

SIGCOMM 2011

Outline

Introduction Datasets and Metrics Analysis Techniques Engagement

View Level Viewer Level

Lessons Conclusion

Introduction

Internet video has become more and more popular

What impacts engagement?! Not well understood yet

Introduction

Given the same video, does Quality impact Engagement?! What are the most critical metrics? Do these critical metrics differ across

genres? How much does optimizing a metric

help?

Datasets and Metrics

Data Collection A week of data from multiple premium

video sites & full census measurement from video player

Video Genres Live LVoD SVoD

Datasets and Metrics

Quality Metrics Buffering Ratio Rate of Buffering Join time Rendering Quality Average Bit Rate

Datasets and Metrics

Two Engagement Granularities View

Play time of a video session Viewer

Total play time by a viewer in a period of time

Total number of views by a viewer in a period of time

Analysis Techniques

Which metrics matter most

Are metrics independent?

How do we quantify the impact?

Analysis Techniques

Qualitative Correlation Coefficient Information Gain

Linear Regression Quantitative

Analysis Techniques

An simple example

View Level Engagement

Long VoD Content - Correlation

View Level Engagement

Long VoD Content - Correlation Most important metric

Buffering ratio

Less important metrics Rendering quality, Join time

View Level Engagement

Long VoD Content – Information Gain

View Level Engagement

Long VoD Content – Information Gain Bit rate becomes the most important

metric Why??????

View Level Engagement

Live Content

View Level Engagement

Live Content Buffering Ration remains the most

significant Bitrate and Rate of Buffering matter

much more

View Level Engagement

Live Content Rendering Quality negatively

correlated?!

User behavior matters

View Level Engagement

Short VoD Content

View Level Engagement

Short VoD Content Similar to long VoD content Buffering ration remains the strongest Rendering Quality is less important

View Level Engagement

Quantitative Impact Not apply regression to all the data Only apply regression to the segment

that looks like linear 0-10% range of Buffering ratio

View Level Engagement

Summary BufRatio is the most important quality metric. For live content, AvgBitrate in addition to

BufRatio is a key quality metric. A 1% increase in BufRatio can decrease 1 to 3

minutes ofviewing time. JoinTime has significantly lower impact on

view-level engagement than the other metrics RendQual in live video highlights the need of

considering context of actual user and system behavior

Viewer Level Engagement

Buffering ratio vs. play time

Viewer Level Engagement

Buffering ratio vs. # of views

Viewer Level Engagement

Summary Both the # of views and the total play

time are impacted by the quality metrics Correlation between the engagement

metrics and the quality metrics becomes visually and quantitatively more striking at the viewer level

The join time, which seemed less relevant at the view level, has non-trivial impact at the viewer level

Lesson Learned

The need for complementary analysis All of you are right. The reason every one of

you is telling it differently is because each one of you touched a different part of the elephant. So, actually the elephant has all the features you mentioned.

Combination of Correlation and Information gain

Lesson Learned

The importance of context Lies, damned lies, and statistics

Together with the context of the human and operating factors

Lesson Learned

Toward video quality index Provide objective index for service

providers and researchersex: MOS

More dimensions More play type More Content type Etc…

Conclusion