View
18
Download
0
Category
Preview:
Citation preview
Spring 2020: Venu: Haag 315, Time: M/W 4-5:15pm
ECE 5578 Multimedia Communication
Lec 10a: Quality of Experience (QoE)
Zhu LiDept of CSEE, UMKC
Office: FH560E, Email: lizhu@umkc.edu, Ph: x 2346.http://l.web.umkc.edu/lizhu
Z. Li: ECE 5578 Multimedia Comm, 2020 p.1
slides created with WPS Office Linux and EqualX LaTex equation editor
Outline
QoE QoS and QoE Def and Standards Body Subjective QoE evaluation – MOS Objective QoE metrics Perceptual QoE metrics
Summary
Z. Li: ECE 5578 Multimedia Comm, 2020 p.2
Video Communication System Coverage
Tentative Topics: QoE Metrics: Referenced, Light Reference, and Reference-less QoE
metrics MPEG Systems: File Format (MP4Box), Streaming Solution
(DASH.js), MMT Media Transport: RTP/RTSP, HTTP/WebSocket, WebRTC, and QUIC Congestion Measure and Modeling in Media Networking Video over Multiple Access Networks : Resource Pricing Solution,
DP+Lagrangian Framework FEC P2P Systems Content Identification and Info Centric Networking
Z. Li: ECE 5578 Multimedia Comm, 2020 p.3
QoS
QoS – Quality of Service A network centric metric Measuring the delay, loss, throughput, ..etc Does not directly translate into user experiences Typically characterized by the packet arrival and departure curves Buffer size: b(A,D, t), delay, d(A,D, t)
Z. Li: ECE 5578 Multimedia Comm, 2020 p.4
QoE
Quality of Experience A user centric metric, how a piece of audio/visual signal delivered by
the network looks/feels Usually a function of QoS, content, and viewing conditions
Fig credit: Touradj Ebrahimi, EPFL, ACM MM 2009
Z. Li: ECE 5578 Multimedia Comm, 2020 p.5
ITU STRUCTURE
3 Sectors:
• Standardization (ITU-T): promotes enabling technical, policy and regulatory frameworks to boost ICT development
• Radiocommunication (ITU-R): coordinates the shared global use of radio spectrum and geostationary satellite orbit
• Development (ITU-D): works to improve telecommunication infrastructure in the developing world
Z. Li: ECE 5578 Multimedia Comm, 2020 p.6
ITU-T Structure and organization
WTSA
TSAG
Study Group x
Working Party 1/x
Question 1/1
Working Party 2/x
Question 1/2
Working Party 3/x
Question 1/3
Study Group y
Working Party 1/y
Question 1/1
Working Parties …
Study Groups …
Z. Li: ECE 5578 Multimedia Comm, 2020 p.7
Study Group 9 OverviewLead Study Group on integrated broadband cable and
television networksResponsible for studies relating to: use of telecommunication systems for contribution, primary
distribution and secondary distribution of television, sound programmes and related data services including interactive services. use of cable and hybrid networks, primarily designed for television
and sound programme delivery to the home, as integrated broadband networks to also carry voice or other time-critical services, video on demand, interactive services, etc.
Z. Li: ECE 5578 Multimedia Comm, 2020 p.8
SG9 QoE metrics work
Rec. # Name Qu Title Timing
J.249 J.redref Q2/9 Perceptual video quality measurement techniques for digital cable television in the presence of a reduced reference
2009
J.340 J.ra-psnr Q2/9 Reference Algorithm for Computing Peak Signal to Noise Ratio (PSNR) of a Video Sequence with Constant Spatial Shifts and a Constant Delay
2009
J.341 J.vqhdtv-fr Q2/9 Objective perceptual multimedia video quality measurement of HDTV for digital cable television in the presence of a full reference
2010
J.bitvqm J.bitvqm Q12/9 Hybrid perceptual bitstream video quality assessment
2013
J.av-dist J.av-dist Q12/9 Methods for subjectively assessing audiovisual quality of internet video and distribution quality television, including separate assessment of video quality and audio quality
2013
J.3D-disp-req
J.3D-disp-req
Q2/9 Display requirements for 3D video quality assessment
2013
Z. Li: ECE 5578 Multimedia Comm, 2020 p.9
Study Group 12 Overview
‘Performance, QoS and QoE’Responsible for Recommendations on performance, quality of
service (QoS) and quality of experience (QoE) for the full spectrum of terminals, networks and services ranging from speech over fixed circuit-based networks to multimedia applications over networks that are mobile and packet based. Included in this scope are the operational aspects of performance, QoS and QoE.
A special focus is given to interoperability to ensure end-to-end users' satisfaction.
SG 12 is the Lead SG on QoS and Performance
Z. Li: ECE 5578 Multimedia Comm, 2020 p.10
SG12 Visual Quality Assessment
Rec. # Name Qu Title Timing
P.1201 P.NAMS Q14/12 Parametric non-intrusive assessment of audiovisual media streaming quality
2012-09
P.1202 P.NBAMS Q14/12 Parametric non-intrusive bitstream assessment of video media streaming quality
2012-09
G.1080 G.IPTV-QoE
Q13/12 Quality of experience requirements for IPTV services
2008
G.1050 Q13/12 Network model for evaluating multimedia transmission performance over the Internet Protocol
2011
G.OMVAS G.OMVAS Q13/12 Opinion model for video and audio streaming applications
2014
P.1401 P.STAT Q9/12 Methods, metrics and procedures for statistical evaluation, qualification and comparison of objective quality prediction models
2012
Z. Li: ECE 5578 Multimedia Comm, 2020 p.11
Video Quality Experts GroupFounded 1997 ITU-T SG 12, SG 9, and ITU-R 11E (now 6C) experts Web ( www.vqeg.org );
First VQEG meeting (Turin, Italy 1997)
Primary mission: Advance the field of video quality
assessment by investigating new and advanced subjective and objective measurement techniques
VQEG does not develop or publish standards Conducts tests and reports results to
ITU and other standards organizations Tests are conducted using specifically
defined procedures (i.e., carefully developed test plans).
Z. Li: ECE 5578 Multimedia Comm, 2020 p.12
VQEG ProjectsCompleted
FRTV I & II (5 ITU Recommendations) Multimedia I (7 ITU Recommendations) RRNR (3 ITU Recommendations) HDTV I (2 ITU Recommendations)
Active: 3DTV (3 ITU Recommendations in progress) Joint Effort Group—JEG-Hybrid Hybrid Perceptual/Bitstream (3 ITU Recommendations in progress) Multimedia II—MM2 (1 ITU Recommendation in progress) Quality for Recognition Tasks—QART (Public Safety, Surveillance
Applications) (1 ITU Recommendation)Ramping up
High Dynamic Range Video—HDR HDTV Phase II—HDTV2 Monitoring of Audio Visual Quality by Key Indicators—MOAVI Real-Time Interactive Communications Evaluation—RICE
Z. Li: ECE 5578 Multimedia Comm, 2020 p.13
VQEG/Standardization Process
ITU-T &SG9, SG12, SG16
ITU-R , WP6C
Other Standards OrgsATIS, IEEE, ETSI, MPEG
VQEG
Results
Industry & Academia
Standards & Reports
Z. Li: ECE 5578 Multimedia Comm, 2020 p.14
Useful Linkshttp://www.itu.int/ITU-T/index.htmlhttp://www.itu.int/ITU-T/studygroups/com09/index.asphttp://www.itu.int/ITU-T/studygroups/com12/index.asphttp://www.itu.int/en/ITU-T/publications/Pages/recs.aspxhttp://www.itu.int/ITU-R/index.asp?category=study-
groups&rlink=rwp6c&lang=en
VQEG: http://www.vqeg.org
Z. Li: ECE 5578 Multimedia Comm, 2020 p.15
Outline
About the Project Sign UpQoE QoS and QoE Def and Standards Body Subjective QoE evaluation – MOS Objective QoE metrics Perceptual QoE metrics
Summary
Z. Li: ECE 5578 Multimedia Comm, 2020 p.16
QoE Subjective Evaluation
MOS – Mean Opinion Score, an user study based quality evaluation
A subjective tests aiming at producing MOS is a delicate mixture of ingredients and choices:• Test/lab environment• Test material• Test methodology• Analysis of the data
credit: Touradj Ebrahimi, EPFL, ACM MM 2009
Z. Li: ECE 5578 Multimedia Comm, 2020 p.17
Test/lab environmentType of Monitors/Speakers and other test equipments
Lighting /Acoustic conditions
Laboratory architecture, background, …
Viewing distance /Hearing position
…
Z. Li: ECE 5578 Multimedia Comm, 2020 p.18
Test material
Meaningful content for the envisaged scenario/application Typical content Worst case content …
p01 p06 p10 bike cafe woman
Z. Li: ECE 5578 Multimedia Comm, 2020 p.19
Test methodology (I)
Single Stimulus (SS)
Non-categorical adjectival or numerical grading scale
5 Excellent 4 Good
3 Fair
2 Poor
1 Bad
5 Imperceptible
4 Perceptible but not annoying
3 Slightly annoying
2 Annoying
1 Very annoying
100
0
Excellent
Bad
Categorical adjectival grading scale: Categorical numerical grading
scale:
“Rate from 1 to 11”
Z. Li: ECE 5578 Multimedia Comm, 2020 p.20
Test methodology (II)
Double Stimulus Impairment Scale (DSIS) Categorical Impairment Grading Scale
5 Imperceptible
4 Perceptible but not annoying
3 Slightly annoying
2 Annoying
1 Very annoying
Z. Li: ECE 5578 Multimedia Comm, 2020 p.21
Test methodology (III)
Double Stimulus Continuous Quality Scale (DSCQS)
Sample 1 Sample 2
Non-categorical adjectival or numerical grading scale:
Sample 1 Sample 2
100
0
Excellent
Bad
100
0
Excellent
Bad
Z. Li: ECE 5578 Multimedia Comm, 2020 p.22
Test methodology (IV)
Stimulus Comparison (SC) Categorical adjectival comparison scale:
“same or different”
much worse
worse
slightly worse
the same
slightly better
better
much better
Non-categorical judgement:
Much worse
Much better
Z. Li: ECE 5578 Multimedia Comm, 2020 p.23
Test methodology (V)
Single Stimulus Continuous Quality Evaluation (SSCQE)
(Very annoying)
(Imperceptible)
Z. Li: ECE 5578 Multimedia Comm, 2020 p.24
Test methodology (VI)
Simultaneous Double Stimulus for Continuous Evaluation (SDSCE)
(Much better)
(Much worse)(Reference) (Test sequence)
Z. Li: ECE 5578 Multimedia Comm, 2020 p.25
Analysis of the MOS data
Improve MOS data quality• Scores distributions across subjects (testing people) is assumed
to be close to normal distribution• Outlier detection and removal• Mean Opinion Scores (MOS) and 95% confidence intervals (CIj)
Nm
MOSN
i ijjå == 1
NNtCI j
j
sa ×-= ),2/1(
mij = score by subject i for test condition j.
N = number of subjects after outliers removal.
t(1-α/2,N) = t-value corresponding to a two-tailed t-Student distribution with N-1 Degrees of Freedom (DoF) and a desired significance level α (α=0.05 in our case, 95% confidence).
σ j = s t a n d a r d d e v i a t i o n o f t h e s c o r e s distribution across subjects for test condition j.
Z. Li: ECE 5578 Multimedia Comm, 2020 p.26
NN
MOSMOStBA
BAobs 22 ss +
-=
What is behind a MOS?
JPEG Image Quality Assessment Study:
Z. Li: ECE 5578 Multimedia Comm, 2020 p.27
Relationship between estimated mean values
• Hypothesis test to find out whether the difference between two MOS values are statistically significant
Two-sided t-test:
• T-statistic:
• Decision rule to reject H0:
BA MOSMOSH =:0
BAa MOSMOSH ¹:
NN
MOSMOStBA
BAobs 22 ss
+
-=
),2/1(),2/( NttORNtt obsobs aa -><
Z. Li: ECE 5578 Multimedia Comm, 2020 p.28
MOS hypothesis test
JPEG
200
0 4:
2:0
JPEG
20
00
4:4:
4JP
EG
JPEG
XR
MS
JPEG
XR
PSJP
EG 2
000
4:2:
0JP
EG
2000
4:
4:4
JPEG
JPEG
XR
MS
JPEG
XR
PSJP
EG 2
000
4:2:
0JP
EG
2000
4:
4:4
JPEG
JPEG
XR
MS
JPEG
XR
PS
JPEG 2000 4:2:0
JPEG 2000 4:4:4JPEG
JPEG XR MS
JPEG XR PS
JPEG 2000 4:2:0
JPEG 2000 4:4:4JPEG
JPEG XR MS
JPEG XR PS
0.25 bpp
0.50 bpp
0.75 bpp
1.00 bpp
1.25 bpp
1.50 bpp
6
5
4
3
2
1
0Number of times
H0 is rejected
Z. Li: ECE 5578 Multimedia Comm, 2020 p.29
Outline
About the Project Sign UpQoE QoS and QoE Def and Standards Body Subjective QoE evaluation – MOS Objective QoE metrics Perceptual QoE metrics
Summary
Z. Li: ECE 5578 Multimedia Comm, 2020 p.30
Objective QoE metrics
• Subjective tests are time consuming, expensive, and difficult to design
• Objective algorithms, i.e. metrics, estimating subjective MOS with high level of correlation are desired• Full reference metrics
• No reference metrics
• Reduced reference metrics
Input/Reference signal
Output/Processed signal
signalprocessing
FR METRIC
Input/Reference signal
Output/Processed signal
signalprocessing
Input/Reference signal
Output/Processed signal
signalprocessing
Features extraction
RR METRIC
Z. Li: ECE 5578 Multimedia Comm, 2020 p.31
PSNR - Peak Signal to Noise Ratio
PSNR def:
Widely used because of its simplicity and ease in formalizing optimization problems!
For image and video data (Y component), a correlation of circa 80% reported when compared to subjective MOS evaluation
åå= =
-=M
1y
N
1x
2ba y)](x,Imy)(x,[Im
MN1MSE
where:
M, N = image dimensions Ima , Imb = pictures to compare B= bit depth
Z. Li: ECE 5578 Multimedia Comm, 2020 p.32
PSNR for color images/video
Multiple channel info, several options to compute metric Weighted PSNR
Weigthed MSE:
Weighted Pixel Value PSNR:
WPSNR = w1PSNR1 + w2PSNR2 + w3PSNR3
)MSEwMSEwMSE(w1)(210log
332211
2B
10 ++-
=WPSNR_MSE
WPSNR_PIX( ) ( )[ ]åå
= =
++-++
-= M
y
N
xbbbaaa
B
)y,x(Imw)y,x(Imw)y,x(Imw)y,x(Imw)y,x(Imw)y,x(ImwMN
)(log
1 1
2332211332211
2
10 11210
Z. Li: ECE 5578 Multimedia Comm, 2020 p.33
PSNR for color images/video
PSNR in RGB vs YCbCro n R
component:
bpp (bits/pixel) bpp (bits/pixel) bpp (bits/pixel)
PSN
R(d
B)
o n G component:
o n B component:
PSN
R (d
B)
PSN
R (d
B)
on Y’ component: on Cb component:
on Cr component:
bpp (bits/pixel) bpp (bits/pixel) bpp (bits/pixel)
Z. Li: ECE 5578 Multimedia Comm, 2020 p.34
• MotivationSimulate relevant early HVS components
Referencesignal
Distortedsignal
Quality/DistortionMeasure
ChannelDecomposition
ErrorNormalization
.
.
.
ErrorPooling
Pre-processing
.
.
.
• Key featuresChannel decomposition linear frequency/orientation transforms
Frequency weighting contrast sensitivity function
Masking intra/inter channel interaction
No separation of objects and illuminance !
/1
,
= åå
l kkleE
Standard IQA Model: Error Visibility
Z. Li: ECE 5578 Multimedia Comm, 2020 p.35
• How to define structural information?
• How to separate structural/nonstructural information?
PhilosophyPurpose of human vision: extract structural information
HVS is highly adapted for this purposeEstimate structural information change
Classical philosophy New philosophyBottom-up Top-down
Predict Error Visibility Predict Structural Distortion
Structural Similarity
New Paradigm
Z. Li: ECE 5578 Multimedia Comm, 2020 p.36
Structural Similarity Measurement
The SSIM system: Image is a product of illuminance and object reflectance Try to separate the object structural info from the illuminance Full reference solution, compare image block x with y, have 3
components:o Luminance, contrast, and structure comparison
Z. Li: ECE 5578 Multimedia Comm, 2020 p.37
[1]
[2]
[3]
�(�,�)
�(�,�)
�(�,�)
Luminance Comparison – l(x,y)
Basic Operations Operate on image regions (can be block, or circular) For each channel, compute the region mean and variance for block x
and its reference y:
Luminance comparison function: L=dynamic range, 2B, e.g, 256 for B=8 K1, small const << 1.
Z. Li: ECE 5578 Multimedia Comm, 2020 p.38
�� =1� �
�=1
��� �� =
1� �
�=1
���
�(�, �) =2���� + (���)�
��� +��
� + (���)
Contrast Comparison – c(x,y)
Compare the illuminance dynamic range and behavior of two blocks Based on the variance of the channel
Contrast function:
Z. Li: ECE 5578 Multimedia Comm, 2020 p.39
�(�,�) =2���� + (���)�
��� +��
� + (���)�
�� = �1
�− 1 ��=1
�
(�� −��)���/�
�� = �1
�− 1 ��=1
�
��� −�����
�/�
Structural Comparison – s(x,y)
Luminance subtraction and variance normalized comparison, supposedly removed illuminance factor, compare objects:
Z. Li: ECE 5578 Multimedia Comm, 2020 p.40
�(�,�) =��� + (���)�
���� + (���)�
��� = �1
�− 1 ��=1
�(�� −��)(�� −��) �
SSIM Index
Structural Similarity Measure (SSIM) General factorized form of power a, b, c:
Typically used: a=b=c=1, �� = �� , let
Then:
Z. Li: ECE 5578 Multimedia Comm, 2020 p.41
�� G(�,�) = [�(�,�)��(�,�)��(�,�)�]
�� G(�,�) =(2���� +��)(2��� +��)
(��� +��
� +��)(��� +��
� +��)
�� = (���)� �� = (���)�
Structural Similarity (SSIM) Index in Image Space
i
k
j
x
xi + xj + xk = 0
x - x
O
luminancechange
contrastchange
structuralchange
xi = xj = xk
),(),(),(),( yxyxyxyx sclSSIM ××=
122
12),(
CC
lyx
yx
++
+=
yx
222
22),(
CC
cyx
yx
++
+=
ssss
yx
3
3),(C
Cs
yx
xy
+
+=
sss
yx
[Wang & Bovik, IEEE Signal Processing Letters, ’02][Wang et al., IEEE Trans. Image Processing, ’04]
Z. Li: ECE 5578 Multimedia Comm, 2020 p.42
original image
JPEG2000 compressed
image
absolute error map
SSIM index map
Z. Li: ECE 5578 Multimedia Comm, 2020
43
original image
Gaussian noise
corrupted image
absolute error map
SSIM index map
Z. Li: ECE 5578 Multimedia Comm, 2020
44
original image
JPEG compressed
image
absolute error map
SSIM index map
Z. Li: ECE 5578 Multimedia Comm, 2020
45
MSE=0, MSSIM=1 MSE=225, MSSIM=0.949 MSE=225, MSSIM=0.989
MSE=215, MSSIM=0.671 MSE=225, MSSIM=0.688 MSE=225, MSSIM=0.723
Demo ImagesZ. Li: ECE 5578 Multimedia Comm,
2020
46
MOS(PSNR) MOS(MSSIM)
0.4 0.5 0.6 0.7 0.8 0.9 10
10
20
30
40
50
60
70
80
90
100
MSSIM (Gaussian window, K1 = 0.01, K2 = 0.03)
MO
S
JPEG images JPEG2000 images Fitting with Logistic Function
15 20 25 30 35 40 45 500
10
20
30
40
50
60
70
80
90
100
PSNR
MO
S
JPEG images JPEG2000 images Fitting with Logistic Function
Dataset JP2(1) JP2(2) JPG(1) JPG(2) Noise Blur Error
# of images 87 82 87 88 145 145 145
PSNR 0.934 0.895 0.902 0.914 0.987 0.774 0.881
SSIM 0.968 0.967 0.965 0.986 0.971 0.936 0.944
Validation with MOS Scores SSIM is a better predictor than PSNR
Z. Li: ECE 5578 Multimedia Comm, 2020 p.47
Summary
QoE is an important component in the multimedia communication system Subjective QoE study: User study generate MOS scores Objective Metrics: compare communicated content as pieces of
signals Perceptive Metrics: try to model HVS and have a better approximation
of MOS
Next Class: Reduced Reference, Non-Reference QoE metrics
Z. Li: ECE 5578 Multimedia Comm, 2020 p.48
Recommended