Upload
daniella-bradley
View
212
Download
0
Embed Size (px)
Citation preview
Christian Schmidmer, OPTICOM 1
Subjective Quality Testing -
Voice & Audio
Christian Schmidmer, OPTICOM 2
MOS acc. To P.800• Standardized Listening Test Procedure
acc. to ITU-T P.800ff
• Absolute Category Rating Test (ACR), no comparison to a reference signal (original)
• „How good does it sound?“
• 5-point grading scale‚opinion scale‘
• Averaging over test subjects: MOS‚Mean Opinion Score‘
ExcellentGoodFairPoorBad
54321
Impairment Grade
Christian Schmidmer, OPTICOM 3
P.CONV
• Framework for conversational tests
• Two test persons having a real conversation
• Example scenarios (e.g. order pizza, book a flight…)
… Still a bit vague, many open questions, current research.
Christian Schmidmer, OPTICOM 4
P.835
• Subjective assessment of noise reduction systems
• Similar to P.800• Separate votes for:
– Speech signal– Background activity (e.g. noise)– Overall quality (high correlation with
P.800 results)
Christian Schmidmer, OPTICOM 5
• Standardised assessment procedure for 'small impairments' in audio systems (ITU-R 1994)
• Comparison between reference and test signal
• Very sensitive to subtle distortions• double-blind triple-stimulus with
hidden reference
Subjective Assessment in ITU-R BS.1116
Original
A B
original / coded
coded / original
Christian Schmidmer, OPTICOM 6
• Continuous grading scale with “anchors”
• Subjective Difference Grade (SDG)• Question: „How different do the
samples sound?“
Impairment Grade
Imperceptible 5.0
Perceptible, but not annoying 4.0
Slightly annoying 3.0
Annoying 2.0
Very annoying 1.0 The ITU-R five-grade impairment scale
Subjective Assessment in ITU-R BS.1116
Christian Schmidmer, OPTICOM 7
Subjective Testing of Intermediate Audio Quality
MUSHRA - Multi Stimulus Test with Hidden Reference and Anchorsdeveloped by EBU working group
B/AIMtargets at IAQ ITU-R BS.1534
Christian Schmidmer, OPTICOM 8
MUSHRA Test
Training of Subjects• comparison with CD quality reference
• two low-pass 'anchors' (7kHz, 3.5kHz) incl.
• subjects can randomly access all types of codecs at similar bitrate
Christian Schmidmer, OPTICOM 9
MUSHRA TestScoring Phase• subjects can randomly assess all codecs
under test of similar bitrate at the same time
• two low-pass 'anchors' (7kHz, 3.5kHz) inc..
• comparison with CD-reference, hidden reference inc..• subjects adjust slider, no score involved
• slider mapped to 0..100
Christian Schmidmer, OPTICOM 10
Comparison of Subjective Test Methods
P.800 BS.1116 BS.1534
Reference Not included Hidden and known Hidden and known
Impairments Large..very large Small Large
Main Application
Speech quality Audio quality Intermediate audio quality
Subjects Inexperienced Expert listeners Expert listeners
Reliability Good Excellent Good
Comment Not applicable to music, influenced by a priori knowledge and expectation
Prb. with low quality signals.
Selection of anchors very critical
Christian Schmidmer, OPTICOM 11
Minimum Requirements
• Headphones (not the cheap ones, no ear buds, no 10$ PC speakers, …)
• Reasonably quiet room• Several listeners• Good soundcard• Knowledge
These are the minimum requirements only, sufficient to do some prescreening. To conduct a reliable subjective test,
much more efforts are required!