Christian Schmidmer, OPTICOM 1

Subjective Quality Testing -

Voice & Audio

Christian Schmidmer, OPTICOM 2

MOS acc. To P.800• Standardized Listening Test Procedure

acc. to ITU-T P.800ff

• Absolute Category Rating Test (ACR), no comparison to a reference signal (original)

• „How good does it sound?“

• 5-point grading scale‚opinion scale‘

• Averaging over test subjects: MOS‚Mean Opinion Score‘

ExcellentGoodFairPoorBad

54321

Impairment Grade

Christian Schmidmer, OPTICOM 3

P.CONV

• Framework for conversational tests

• Two test persons having a real conversation

• Example scenarios (e.g. order pizza, book a flight…)

… Still a bit vague, many open questions, current research.

Christian Schmidmer, OPTICOM 4

P.835

• Subjective assessment of noise reduction systems

• Similar to P.800• Separate votes for:

– Speech signal– Background activity (e.g. noise)– Overall quality (high correlation with

P.800 results)

Christian Schmidmer, OPTICOM 5

• Standardised assessment procedure for 'small impairments' in audio systems (ITU-R 1994)

• Comparison between reference and test signal

• Very sensitive to subtle distortions• double-blind triple-stimulus with

hidden reference

Subjective Assessment in ITU-R BS.1116

Original

A B

original / coded

coded / original

Christian Schmidmer, OPTICOM 6

• Continuous grading scale with “anchors”

• Subjective Difference Grade (SDG)• Question: „How different do the

samples sound?“

Impairment Grade

Imperceptible 5.0

Perceptible, but not annoying 4.0

Slightly annoying 3.0

Annoying 2.0

Very annoying 1.0 The ITU-R five-grade impairment scale

Subjective Assessment in ITU-R BS.1116

Christian Schmidmer, OPTICOM 7

Subjective Testing of Intermediate Audio Quality

MUSHRA - Multi Stimulus Test with Hidden Reference and Anchorsdeveloped by EBU working group

B/AIMtargets at IAQ ITU-R BS.1534

Christian Schmidmer, OPTICOM 8

MUSHRA Test

Training of Subjects• comparison with CD quality reference

• two low-pass 'anchors' (7kHz, 3.5kHz) incl.

• subjects can randomly access all types of codecs at similar bitrate

Christian Schmidmer, OPTICOM 9

MUSHRA TestScoring Phase• subjects can randomly assess all codecs

under test of similar bitrate at the same time

• two low-pass 'anchors' (7kHz, 3.5kHz) inc..

• comparison with CD-reference, hidden reference inc..• subjects adjust slider, no score involved

• slider mapped to 0..100

Christian Schmidmer, OPTICOM 10

Comparison of Subjective Test Methods

P.800 BS.1116 BS.1534

Reference Not included Hidden and known Hidden and known

Impairments Large..very large Small Large

Main Application

Speech quality Audio quality Intermediate audio quality

Subjects Inexperienced Expert listeners Expert listeners

Reliability Good Excellent Good

Comment Not applicable to music, influenced by a priori knowledge and expectation

Prb. with low quality signals.

Selection of anchors very critical

Christian Schmidmer, OPTICOM 11

Minimum Requirements

• Headphones (not the cheap ones, no ear buds, no 10$ PC speakers, …)

• Reasonably quiet room• Several listeners• Good soundcard• Knowledge

These are the minimum requirements only, sufficient to do some prescreening. To conduct a reliable subjective test,

much more efforts are required!