Ultra-High-Definition Quality of Experience with MPEG-DASH

Ultra-High-Definition Quality of Experience with MPEG-DASH

Priv.-Doz. Dr. Christian TimmererDaniel Weinberger, Christopher Mueller and Stefan Lederer

Chief Innovation Officer (CIO) at bitmovin GmbHhttp://www.bitmovin.com christian.timmerer@bitmovin.com

Alpen-Adria-Universität Klagenfurt (AAU) Faculty of Technical Sciences (TEWI) Department of Information Technology (ITEC) Multimedia Communication (MMC) Sensory Experience Lab (SELab)

http://blog.timmerer.com http://selab.itec.aau.at/ http://dash.itec.aau.at christian.timmerer@itec.aau.at

http://www.slideshare.net/christian.timmerer

Outline• Introduction• Quality, Quality of Experience, and DASH• Evaluation strategies• Results• Conclusions

• Acknowledgment [some slides]: Ali C. Begen, CISCOApril 16, 2015 NAB2015 - BEC - QoE-DASH 2

Introduction• Real-time entertainment

– Streaming video and audio– > 60% of Internet traffic

• All delivered over-the-top (OTT)• MPEG Dynamic Adaptive Streaming

over HTTP (DASH)– Coding format agnostic

• DASH Industry Forum– Interoperability Points (IOPs) for

common codecs and others (v3.0)– E.g., AVC/H.264, HEVC/H.265, and AAC

April 16, 2015 NAB2015 - BEC - QoE-DASH 3

Over-The-Top – Adaptive Media Streaming

April 16, 2015 NAB2015 - BEC - QoE-DASH 4

Adaptation logic is within the client, not normatively

specified by the standard, subject to research and

development

Open Digital Media Value Chain

April 16, 2015 NAB2015 - BEC - QoE-DASH 5

Create Content

Aggregate

Monetize

Distribute

Content

Consume Content

Any Content Any Storefront

Any Network Any Device

CDNsMedia Protocol

s

Internet Transpor

t

DRMEncoding

Encapsulation

DynamicAds

Clients

Simplified Example Workflow: bitcodin/bitdash

April 16, 2015 NAB2015 - BEC - QoE-DASH 6

Source: http://www.bitmovin.net/bitcodin-cloud-based-transcoding-streaming-platform/

Internet TV vs. Traditional TV in 2010• Areas most important to overall TV

experience are– Content– Timing control– Quality– Ease of use

• While traditional TV surpasses Internet TV only in quality, it delivers better “overall experience”

April 16, 2015 NAB2015 - BEC - QoE-DASH 7

When comparing traditional and Internet TV, which option is better?

Traditional Internet

Content 7% 79%

Timing / Control 7% 83%

Quality 80% 16%

Ease of Use 23% 52%

Control (FF, etc.) 9% 77%

Portability 4% 92%

Interactivity 31% 52%

Sharing 33% 56%

Overall Experience 53% 33%

Source: Cisco IBSG Youth Survey, Cisco IBSG Youth Focus Group Sessions, 2010

Quality (of Experience)• QoE as evolution of QoS [ITU-T

P.10/G.100]• QoS: totality of characteristics of a

telecommunications service that bear on its ability to satisfy stated and implied needs of the user of the service

• QoE: the overall acceptability of an application or service, as perceived subjectively by the end-user

April 16, 2015 NAB2015 - BEC - QoE-DASH 8

Many definitions but in general, it’s like an elephant

Quality of Experience• COST Action IC1003 – QUALINET (http://www.qualinet.eu/)

“the degree of delight or annoyance of the user of an application or service. It results from the fulfillment of his or her expectations with respect to the utility and/or enjoyment of

the application or service in the light of the user’s personality and current state”

• QoE influence factors– Any characteristic of a user, system, service, application, or context– Grouped into human, system, and context

• QoE features– Perceivable, recognized and namable characteristic of the individual’s experience– Depends on the level of direct perception, interaction, the usage situation

April 16, 2015 NAB2015 - BEC - QoE-DASH 9

QoE for DASH• Different application domains have different QoE requirements

– Need to provide specializations of the general QoE definition– Take into account requirements formulated by means of influence

factors and features of QoE• QoE influence factors for DASH

– Initial/start-up delay (low)– Buffer underruns, stalls, freezes (zero)– Quality switches (low)– Media throughput (high)– …

April 16, 2015 NAB2015 - BEC - QoE-DASH 10

QoE Evaluation for DASH-based Services• Test sequence

– Many datasets available– Adopted Big Buck Bunny & DASHed it with bitcodin

• Players– bitdash– …and compare it with ten different adaptation algorithms

• Objective evaluation– Test setup– Predefined bandwidth trajectory (or real network traces)

• Subjective evaluation– Lab vs. crowdsourcing

April 16, 2015 NAB2015 - BEC - QoE-DASH 11

http://www.bitcodin.com/

http://www.dash-player.com/

Objective Evaluations

April 16, 2015 NAB2015 - BEC - QoE-DASH 12

Stalls (lower is better)Average Bitrate (higher is better)

Stalls are really bad…

April 16, 2015 NAB2015 - BEC - QoE-DASH 13

Conviva: Viewer Experience Report. 2014

DASH-JS vs. bitdash

April 16, 2015 NAB2015 - BEC - QoE-DASH 14

Subjective Evaluation• Microworker platform

– Limited to Europe, USA/Canada, India• DASH clients

– DASH-JS (dash.itec.aau.at)– dash.js (DASH-IF)– YouTube

• Tears of Steal trailer according to YouTube configuration

• Screening techniques– Browser fingerprinting– Presentation time– QoE ratings and Pre-QuestionnaireApril 16, 2015 NAB2015 - BEC - QoE-DASH 15

What about 4K and 8K?

April 16, 2015 NAB2015 - BEC - QoE-DASH 16

• Why? – because we can!• Supported on the Web

– HTML5, MSE– AVC/H.264– [HEVC/H.265 needed

to lower bitrate]• See demo @

http://www.dash-player.com/

• UHD-QoE evaluation

Conclusions• QoE for DASH-based services (a rule of thumb)

– Startup delay (low [but live vs. on-demand & short vs. long-tail content])– Buffer underrun / stalls (zero)– Quality switches (low) and media throughput (high)– Energy- and cost-awareness (data plan)

• No general applicable QoE model for DASH– (Too) many factors influencing / features of QoE for DASH-based services– Methodology for reproducible research is in place and well established– Ample research opportunities

April 16, 2015 NAB2015 - BEC - QoE-DASH 17

Main QoE factors for DASH

Thank you!

April 16, 2015 NAB2015 - BEC - QoE-DASH 18

Source: http://www.bitmovin.net/bitcodin-cloud-based-transcoding-streaming-platform/