ConférenSquad #3 : Subjective Quality and HTTP Adaptive Streaming: a Review of Psychophysical Studies (Frédéric Dufaux, Telecom Paris Tech)

Institut Mines-Télécom

Subjective quality and HTTP adaptive streaming: a review of psychophysical studies

Francesca De Simone, Frédéric DufauxInstitut Mines-Télécom; Télécom ParisTech; CNRS LTC I


Content

� Basic concepts• Quality of Service (QoS) vs Quality of Experience (QoE)• Subjective vs objective assessment• QoE and DASH

� Subjective QoE and DASH: what do we know• Perception of stalling events • Perception of quality switches• Tradeoffs

� Conclusions

2


Basic Concepts

3


QoS vs QoE

� Example of Quality of Service (QoS) metrics• Available bandwidth• Round Trip Time (RTT)• Packet loss rate (PLR)• Etc..

� Quality of Experience (QoE)?

4


QoS vs QoE

� Example of Quality of Service (QoS) metrics• Available bandwidth• Round Trip Time (RTT)• Packet loss rate (PLR)• Etc..

� Quality of Experience (QoE)? [Qualinet WhitePaper 2012]

5

multimedia experiencemultimedia experience

MULTIMEDIA

SERVICEVisual

signal

PriceNovelty

Interface

CONTEXT

etc…

Audio

signal

etc…USER

Human Visual

System

Expectations

Emotions

Auditory

System

Attention

etc…


Subjective vs objective assessment

� Subjective quality assessment� In controlled environment• standard methodologies for video

designed for analog television [ITU Rec BT.500-11]

• adapted for multimedia applications [ITU Rec. P.910]

� In uncontrolled environment• crowdsourcing• field experiments• users’ feedback from real products

6


Subjective vs objective assessment

� Objective quality assessment• Full-Reference (FR)• Reduced-Reference (RR) • No-Reference (NR)

7

Input/Reference signal

Output/Processed signalImage/ video

processing

FR METRIC



processing NR METRIC



processing

Feature

extraction RR METRIC

� Example of standardized solutions• FR video quality metric for HDTV for digital cable television [ITU-T

Rec. J.341]• NR model for audiovisual quality assessment in IPTV [ITU-T Rec.

P.1201.2]


QoE and DASH

HTTP server TCP transport layer DASH client

128 kbps

500kbps

1Mbps

• Terminology

0 2 4 6 8 10 12 14 time (s)

Realization

Realization

Realization

Chunk Adaptation Set

128 kbps

500kbps

1Mbps


QoE and DASH

� No packet losses• Reliable transport layer

� Stalling events• Playout interruptions

due to video player buffer underflow

� Quality switches• Quality variation due to

switch betweenrepresentations

9


QoE and HAS

� No packet losses• Reliable transport layer

� Stalling events• Playout interruptions

due to video player buffer underflow

� Quality switches• Quality variation due to

switch betweenrepresentations

10

Not addressed in traditional

video QoE research!


Goal of this talk

11

� Identify key questions concerning subjective video quality perception and DASH

� Review existing answers• Perception of video stalling• Perception of video quality switches• Tradeoffs


Subjective QoE and DASH: what do we know

12


Perception of stalling events

13

� Stallings should be avoided at all times [Dobrian2011, Mok2011, Floris2012, Atzori2013, Hobfeld2013] but if they occur…

“what is the impact of stalling frequency and duration?”



14

� Stallings should be avoided at all times [Dobrian2011, Mok2011, Floris2012, Atzori2013, Hobfeld2013] but if they occur…

“what is the impact of stalling frequency and duration?”

• Studies before DASH [Pastrana-Vidal2004, Staelens2010]• Studies on DASH [Moorthy2011, vanKester2011]



� Conclusions• Threshold to detect stalling: 80ms [Pastrana-Vidal2004]• Subjects prefer a single stalling of longer duration compared

to multiple short stallings [Moorthy2011]• Subjects prefer regular stallings over irregular ones

[Moorthy2011]• Stallings up to 400ms can still be tolerated in the case of few

stalling events and long video sequences [vanKester2011] [Staelens2010]

15


Perception of quality switches

16

� Assumptions• Each representation corresponds to a

different bitrate• Representations may differ in terms of

spatial resolution, temporal resolution and encoding quantization settings

• Three kinds of switches are possible• Encoding switch (ESW)• Spatial switch (SSW)• Temporal switch (TSW)

128 kbps

500kbps

1Mbps



17

“What is the impact of switching amplitude and frequency?”



18


Question 1: “shall the bitrate be reduced (increased) once with a big quality drop or gradually?”

time

Laye

r o

r

Ra

te

time

Laye

r o

r

Ra

te



19

– Studies on layer encoded videos and SVC [Zink 2003, Ni 2011] – Studies on HAS [Moorthy 2012, Mok 2012, Yitong 2013]


Question 1: “shall the bitrate be reduced (increased) once with a big quality drop or gradually?”

time

Laye

r o

r

Ra

te

time

Laye

r o

r

Ra

te



20

� Conclusions• Smooth variations preferred to abrupt variations• Amplitude of the variation must be kept as small as possible• Examples of amplitude thresholds to deliver generally

acceptable quality [Ni 2011]• Quality differences should not exceed 4 QPs for ESW, one third of

the original frame rate for TSW, half the original size for SSW



21

� But…• Conclusion might not be generalizable to switches between

levels exhibiting small quality separation [Moorthy 2012]

� Conclusions• Smooth variations preferred to abrupt variations• Amplitude of the variation must be kept as small as possible• Examples of amplitude thresholds to deliver generally

acceptable quality [Ni 2011]• Quality differences should not exceed 4 QPs for ESW, one third of

the original frame rate for TSW, half the original size for SSW



22

Question 2: “Do users prefer shorter but more frequent variation or longer but less frequent variations?”



23

• [Zink 2003, Ni 2011] layer encoded videos and SVC

• [Moorthy 2012, Robinson 2012, Yitong 2013]

Question 2: “Do users prefer shorter but more frequent variation or longer but less frequent variations?”

time

Laye

r o

r

Ra

te

time

Laye

r o

r

Ra

te

Institut Mines-Télécom24

� Conclusions• Less frequent quality variations are preferred to more frequent

variations• Frequency of variations should be kept as small as possible

• For SNR or spatial resolution switches, low frequency can relieve the annoyance of quality switch [Ni 2011]


Institut Mines-Télécom25

� Conclusions• Less frequent quality variations are preferred to more

frequent variations• Frequency of variations should be kept as small as

possible• For SNR or spatial resolution switches, low frequency can

relieve the annoyance of quality switch [Ni 2011]

� But…• More frequent switches are preferred over fewer

switches, if the subject is able to view the highest quality video for longer duration [Moorthy 2012]

• For temporal resolution switches, the frequency does not seem to have significant influence on QoE [Ni 2011]




26

Question 3: “is it better to switch quality levels or to stay at a lower quality? i.e. to switch or not to switch?”



27

Question 3: “is it better to switch quality levels or to stay at a lower quality? i.e. to switch or not to switch?”

• [Zink 2003, Ni 2011] layer encoded videos and SVC

• [Moorthy 2012, Robinson 2012, Mok 2012, Yitong2013, Villa 2013, Rehman2013, Tavakoli 2014]

time

Laye

r o

r

Ra

te

time

Laye

r o

r

Ra

te



28

� Conclusions• Constant quality preferred to varying quality

• Short-term spikes degrade QoE [Yitong 2013] • Constant (lower) quality better than decreasing quality (from higher

to lower) [Zink 2003] • Constant or nearly constant quality is preferable to frequently

varying quality (even if mean quality is lower) [Robinson 2012] [Ni 2011]

• Providing as high bitrate as possible does not necessarily lead to the highest QoE [Mok 2012]



29

� But…• Preferable to switch to a higher rate, if the duration of the

higher rate is at least half the duration of the lower rates [Moorthy 2012]

• If constant quality is too low, any adaptation is preferred [Tavakoli 2014] [Ni 2011]

• Maintaining a reasonable quality for longer duration results in a small bias towards better subjective experience [Rehman 2013]



30

� Temporal trend• The end quality of the video makes a definite impact on the perceived

quality: all is well that ends well [Zink 2003][Moorthy 2012]• [TavakoliSPIE2014] For increasing quality, overall results did not show

any statistically significantly difference between different strategies. For the decreasing scenarios, there was a statistically significantly preference to gradual change with 10 seconds chunk compared to the other scenarios.

� Content dependency• [NiMMsys2011] [RobinsonBellTJ2012] Effect of switch of spatial and

temporal resolution varies on different content types, even for the same amplitude• difficult to spot quality oscillations when frequent scene changes in the

scene. • Quality change is more noticeable in steady, pan, tracking shots and when

there are number of hard edges. • More complex texture details , more strongly affected by the loss of spatial

resolution• The lack of relative movement between objects in the scene may limit the

visible effect of frame dropping.



When the bitrate has to be lowered (increased), for instance due to restricted (better) network conditions, …� Question 1: “is it better to reduce (increase)

encoded bitrate, frame rate, or resolution?”

31



32

When the bitrate has to be lowered (increased), for instance due to restricted bandwidth (better networ k conditions), …� Question 1: “is it better to reduce (increase) the

encoded bitrate, the video frame rate, or the video resolution?”

H.264/SVC encoded versions at same bitrate

1/16 of the original resolution upscaledStronger quantization

• Many subjective studies on optimal combinations (SNR, spatial resolution, temporal resolution) for scalable video coding [Lee 2012]



33

� Conclusions

• Spatial quality more important than frame rate at low bit rates and viceversa at high bit rates

• Low vs high bit rate threshold higher for content with faster motion

• Among quality switch of SNR levels, spatial resolutions or temporal resolutions, SNR switch is the most recommended as less noticeable [Ni 2011]



34

� Methodological findings:• [RehmanQomex2013] Subjects are resistent in updating their

opinions: when there is a small quality variation between consecutive scenes, subjects tend to keep their opinions or change only slightly.

• [RehmanQomex2013] [ChenIP2014] Subjects use asymmetric strategies in updating their opinions. A significant quality degradation between consecutive scenes results in a large penalty, as compared to the reward obtained by a similar quality improvement between consecutive scenes. Behavioral response of viewers to quality variation is more sensitive in low quality region than in high quality region.

• [ChenIP2014] The QoE of a video at a particular moment depends on the viewing experience before the moment: the current video quality can affect the perceived quality in the next 15 second

• [ChenIP2014] A viewer suffering a previous unpleasant viewing experience tends to penalize the perceived quality in the future

• [VillaNIK2013] Significant differences in terms of self-reported pleasure, arousal, and the degree of delight between the two usage scenarios (user alone vs group of users). All significantly higher in the first usage scenario.


Tradeoffs: initial delay vs stalling

35

� Stallings can be avoided by employing larger client buffers but this will affect the initial sta rtup delay of the video, so…

� Question 2: “what is the impact of initial loading?”� Question 3: “what is the impact of combined effect

(stalling + initial loading)?”



36

� Stallings can be avoided by employing larger client buffers but this will affect the initial sta rtup delay of the video, so…

� Question 2: “what is the impact of initial loading?”� Question 3: “what is the impact of combined effect

(stalling + initial loading)?”

� [Hobfeld2013, HoBfeldQomex2012, Staelens2010]



� Conclusions• Initial startup delay does not significantly influence

quality perception [Hoßfeld 2013]• End-users are willing to tolerate larger startup delays,

if this results in less video stallings [Staelens 2010]• Different amounts of startup delay are tolerated,

depending on the specific type of application [HoBfeld 2012]

37


Conclusions

38


Conclusions

� Agreed upon conclusions:• Frequency of stalling is the main factor affecting QoE• The amplitude of the quality switch is the most

dominant factor for the perception of quality switch (more than the frequency)

� Issue:• How generalizable are the findings of these studies?

39


Challenges

40

� Why is it difficult to design a subjective test for HAS?• Temporal variation of quality

─ Longer sequences may be needed: which test method to use?

• Variety of terminals in the real scenario─ Different network conditions─ Different users’ expectations


Thank You for your Attention!francesca.de-simone@[email protected]

Thanks to:Marie-Neige Garcia and Alexander Raake (T-Labs TUBerlin), Nicolas Staelens (Ghent University), SamiraTavakoli (Universidad Politecnica de Madrid), SebastianEgger (ftw), Kjell Brunnstrom (Acreo)

41


Further reading and resources…

1. Qualinet White Paper on Definitions of Quality of Ex perience (2012), European Network on Quality of Exp erience in Multimedia Systems and Services (COST Action IC 100 3), Patrick Le Callet, Sebastian Möller and Andrew P erkis, eds., Lausanne, Switzerland, Version 1.1, June 3, 2012.

2. Zink et al., Subjective impression of variations in layer encoded videos, ACM 20033. R.R. Pastrana-Vidal, J.C. Gicquel, C. Colomes, and H. Cherifi, “Sporadic frame dropping impact on qual ity perception,” in

Proceedings of SPIE , 2004, vol. 5292, p. 1824. Q. Huynh-Thu and M. Ghanbari, “Temporal Aspect of P erceived Quality in Mobile Video Broadcasting”, IEE E Transactions on

Broadcasting, vol.54, no.3, pp.641-651, Sept. 20085. J.-S. Lee, F. De Simone, N. Ramzan, E. Izquierdo an d T. Ebrahimi, Quality assessment of multidimensional video scalability,

in IEEE Communications Magazine, vol. 50, num. 4, p . 38-46, 20126. J.-S. Lee, F. De Simone and T. Ebrahimi, Subjective quality evaluation via paired comparison: applicat ion to scalable video

coding, in IEEE Transactions on Multimedia, vol. 13 , num. 5, p. 882-893, 20117. Venkataraman et al., Effects of Internet path select ion on video QoE, MMSys 20118. Chen et al., Quantifying QoS requirements of network services: a cheat-proof framework, MMSys 20119. R. Mok, E. Chan, and R. Chang. Measuring the qualit y of experience of HTTP video streaming. In Proc. I EEE/IFIP10. IM (pre-conf.), 201111. R. Mok, E. Chan, X. Luo, and R. Chang., Inferring t he QoE of HTTP Video Streaming from User-Viewing Act ivities, ACM W-

MUST’11, 201112. Zhou Wang; Bovik, A.C., "Mean squared error: Love i t or leave it? A new look at Signal Fidelity Measur es," Signal Processing

Magazine, IEEE , vol.26, no.1, pp.98-117, Jan. 200913. Kusching at el., An evaluation of TCP-based rate-con trol algorithms for adaptive internet streaming of H.264/SVC, MMSys

201014. Lederer at el., Dynamic adaptive streaming over HTTP dataset, MMSys 201215. K. Seshadrinathan and A. C. Bovik, Motion Tuned Spat io-temporal Quality Assessment of Natural Videos, v ol. 19, no. 2, pp.

335-350, IEEE Transactions on Image Processing , Feb. 201016. NTIA VQM http://www.its.bldrdoc.gov/resources/video-quality- research/software.aspx17. Yim et al., Evaluation of temporal variation of vide o quality in packet loss network, Signal Processing Image Communication,

201118. Singh, K.D.; Hadjadj-Aoul, Y.; Rubino, G.; , "Quali ty of experience estimation for adaptive HTTP/TCP v ideo streaming using

H.264/AVC," Consumer Communications and Networking Conference (CCNC), 2012 IEEE , vol., no., pp.127-13 1, 14-17 Jan. 2012

42

Technology

ConférenSquad #3 : Subjective Quality and HTTP Adaptive Streaming: a Review of Psychophysical Studies (Frédéric Dufaux, Telecom Paris Tech)