Embed Size (px)
Citation preview
Department of Communication and Electronic EngineeringUniversity of Plymouth, U.K.
Lingfen SunEmmanuel Ifeachor
New Methods for Voice Quality Evaluation for IP Networks
4 September, 2003 ITC-18, Berlin, Germany 2
Outline
Introduction Subjective MOS test from traditional to
Internet based Intrusive voice quality measurement from
listening-only to conversational Non-intrusive measurement from subjective
based to objective based Conclusions
4 September, 2003 ITC-18, Berlin, Germany 3
Introduction
Aims: to investigate new subjective and objective measurement methods for VoIP applications
Subjective tests MOS (Mean Opinion Score), user perceived quality Benchmarking objective methods Slow, time consuming and expensive
Objective measurements Intrusive methods (e.g. PESQ, only listening quality) Non-intrusive methods (e.g. E-model, only for limited
codec/applications)
4 September, 2003 ITC-18, Berlin, Germany 4
Traditional MOS test
Strict test requirement (e.g. sound proof room)
Guarantee consistent testing environment Slow, time-consuming and expensive Test environment far to reality Originated from codec quality assessment
(the quality difference is subtle).
4 September, 2003 ITC-18, Berlin, Germany 5
Internet-based MOS test
Based on Internet, carried out at office, project room or classroom, …, close to reality
Easier access to large number of subjects (e.g. 39 tests at the same time)
Save time and money Lack of a controlled testing environment
(e.g. background noise).
4 September, 2003 ITC-18, Berlin, Germany 6
Online MOS test
4 September, 2003 ITC-18, Berlin, Germany 7
Online MOS test
4 September, 2003 ITC-18, Berlin, Germany 8
Online MOS test results
4 September, 2003 ITC-18, Berlin, Germany 9
Voice Quality Evaluation
Referencespeech
encoder Trace dataJitterbuffer
Degraded speech
PESQ MOS
(PESQ)
E-model MOS (E-model)
decoder
Subjective test MOS
(Web or Room)
4 September, 2003 ITC-18, Berlin, Germany 10
Performance Comparison
MOS Comparison
0
1
2
3
4
5
0 5 10 15 20 25 30
Packet Loss (%)
MO
S
Room_MOS PESQ_MOS Web_MOS E-model_MOS
4 September, 2003 ITC-18, Berlin, Germany 11
Performance Comparison
Results show that Internet-based MOS test compares well with traditional MOS test.
Two objective test methods (PESQ and E-model) can both predict subjective MOS score well.
Correlation coefficients for MOS comparison
Name PESQ vs.
RMOS
PESQ vs.
WMOS
Emodel vs.
RMOS
Emodel vs.
WMOS
WMOS vs.
RMOS
Emodel vs.
PESQ
Corr. Coeff.
0.933 0.984 0.935 0.964 0.952 0.975
4 September, 2003 ITC-18, Berlin, Germany 12
Intrusive conversational quality measurement
PESQ is the latest ITU standard for intrusive voice quality measurement.
It can only predict one-way listening voice quality.
How to extend it to predict conversational quality? One solution: to combine with E-model
4 September, 2003 ITC-18, Berlin, Germany 13
Procedures: Obtain MOS from PESQ based on a comparison of reference
and degraded speech Convert it back to R-factor, then to Ie Obtain Id from end-to-end delay Calculate MOSc from Id and Ie from E-model
Intrusive conversationalmeasurement
PESQ
Delay model
MOS R Ie
Ie
End-to-end delay
E-modelMOSc
Id
Reference speech
Degraded speech
MOS (PESQ)
4 September, 2003 ITC-18, Berlin, Germany 14
R-factor with MOS
0.5 1 1.5 2 2.5 3 3.5 4 4.5 50
20
40
60
80
100
MOS
R-v
alu
e
R-value vs. MOS
from G.1073rd order polynomial fitting
100 5.4
1000 107)100)(60(035.01
0 16
RforMOS
RforRRRRMOS
RforMOS(1)
336.57060.872314.253026.3 MOSMOSMOSR (2)
( From R to MOS, from G.107)
( From MOS to R, by 3rd order polynomial)
4 September, 2003 ITC-18, Berlin, Germany 15
Combined with E-model
RIe 2.93
0 if 1)(
0 if 0)(
)3.177()3.177(11.0024.0
xxH
xxHwhere
THTTI aaad
(3)
(4)
ed IIR 2.93 (5)
Convert R back to MOS to obtain conversational voice quality
4 September, 2003 ITC-18, Berlin, Germany 16
Non-intrusiveobjective measurement
E-model is the latest ITU standard for non-intrusive voice quality prediction.
Its parameters/equations (e.g. Ie) are still based on subjective tests.
Difficult to apply to new codecs/applications How to improve it ?
Based on objective methods (e.g. PESQ) to derive parameters/equations for new codecs and applications.
4 September, 2003 ITC-18, Berlin, Germany 17
Procedure 1:obtain MOS (PESQ)
Obtain MOS (PESQ) vs. packet loss rate (random) for the codec (e.g. AMR 12.2 Kb/s)
MOS (PESQ) value was obtained by averaging over different speech samples (male and female) in ITU data base and different loss locations/patterns.
0 5 10 15 20 25 301.5
2
2.5
3
3.5
4
4.5MOS vs. Packet loss rate for AMR (12.2Kb/s)
Packet Loss (random loss, %)
MO
S(P
ES
Q)
4 September, 2003 ITC-18, Berlin, Germany 18
Procedure 2:Obtain Ie equation
Convert MOS vs. packet loss to Ie vs. packet loss using Equations (2) and (3).
Curve fitting to obtain equation of Ie vs. loss.
0 5 10 15 20 25 3010
15
20
25
30
35
40
45
50
55Ie vs. Packet loss rate for AMR (12.2Kb/s)
Packet Loss (random loss, %)
Ie
from PESQfrom fitting
)*38.01ln(84.152.13 lossIe (6)
4 September, 2003 ITC-18, Berlin, Germany 19
Procedure 3: Obtain MOS (loss, delay)
From Ie and Id, R can be derived, then MOS can be obtained directly from packet loss and end-to-end delay.
This can be used for non-intrusive quality monitoring and perceptual buffer optimization and perceived QoS control.
0
200400
0 5 10 15 20 25 30
0.5
1
1.5
2
2.5
3
3.5
4
4.5
Delay (ms)
MOS vs. packet loss and delay (for AMR,12.2Kb/s)
Packet loss (%)
MO
S
4 September, 2003 ITC-18, Berlin, Germany 20
Conclusions/Future work
Investigated novel subjective and objective voice quality evaluation methods.
Future work More extensive Internet-based MOS test
and comparison with P.800 MOS test New applications for predicted perceived
voice quality.
4 September, 2003 ITC-18, Berlin, Germany 21
Contact
MOS Test Website:http://www.tech.plymouth.ac.uk/spmc/people/lfsun/mos/ Contact
Lingfen Sun: [email protected] Emmanuel Ifeachor: [email protected]
Thank you!
