Doc.: IEEE 802.11-06/0965r1 Submission July 2006 Royce Fernald - Intel CorporationSlide 1 Video Performance Notice: This document has been prepared to

July 2006

Royce Fernald - Intel Corporation

Slide 1

doc.: IEEE 802.11-06/0965r1

Submission

Video Performance

Notice: This document has been prepared to assist IEEE 802.11. It is offered as a basis for discussion and is not binding on the contributing individual(s) or organization(s). The material in this document is subject to change in form and content after further study. The contributor(s) reserve(s) the right to add, amend or withdraw material contained herein.

Release: The contributor grants a free, irrevocable license to the IEEE to incorporate material contained in this contribution, and any modifications thereof, in the creation of an IEEE Standards publication; to copyright in the IEEE’s name any IEEE Standards publication even though it may include portions of this contribution; and at the IEEE’s sole discretion to permit others to reproduce in whole or in part the resulting IEEE Standards publication. The contributor also acknowledges and accepts that this contribution may be made public by IEEE 802.11.

Patent Policy and Procedures: The contributor is familiar with the IEEE 802 Patent Policy and Procedures <http:// ieee802.org/guides/bylaws/sb-bylaws.pdf>, including the statement "IEEE standards may include the known use of patent(s), including patent applications, provided the IEEE receives assurance from the patent holder or applicant with respect to patents essential for compliance with both mandatory and optional portions of the standard." Early disclosure to the Working Group of patent information that might be relevant to the standard is essential to reduce the possibility for delays in the development process and increase the likelihood that the draft publication will be approved for publication. Please notify the Chair <[email protected]> as early as possible, in written or electronic form, if patented technology (or technology under patent application) might be incorporated into a draft standard being developed within the IEEE 802.11 Working Group. If you have questions, contact the IEEE Patent Committee Administrator at <[email protected]>.

Name Company Address Phone Email Royce Fernald Intel 5200 Elam Young Pkwy

MS:HF3-96 Hillsboro, OR 97124

503-696-4318 [email protected]

Uriel Lemberger

Intel Matam Industrial Park PO Box 1659 Haifa 31015 Israel

+972-4-865-5701

[email protected]

Alexander Tolpin Intel Matam Industrial Park PO Box 1659 Haifa 31015 Israel

+972-4-865-5430

[email protected]

Philip Corriveau Intel 5200 Elam Young Pkwy MS:HF3-96 Hillsboro, OR 97124


Fahd Pirzada Dell One Dell Way Round Rock, TX 78682


Pratik Mehta Dell One Dell Way Round Rock, TX 78682


Fanny Mlinarsky Azimuth 31 Nagog Park, Acton, MA 01720


Dalton Victor Broadcom 190 Mathilda Place Sunnyvale, CA 94025


Mark Kobayhashi Broadcom 190 Mathilda Place Sunnyvale, CA 94025


Authors:

July 2006


Slide 2

doc.: IEEE 802.11-06/0965r1

Submission

Abstract

This presentation defines the primary metrics for video performance as well as key secondary metrics

This presentation corresponds to the draft text submission 802.11-06/0964r1

July 2006


Slide 3

doc.: IEEE 802.11-06/0965r1

Submission

Agenda

Background– Proposal Framework– Test Output– Video Delivery and Video Quality– Measuring Video Delivery and Video Quality– The Media Delivery Index– Video Delivery, Video Quality and Wireless Performance– References

• Test Description (doc.: 802.11-06/0964r1)– Introduction and Purpose– Resource Requirements– Test Setup– Video Content Preparation and Analysis– Calibration– Baseline Configuration– Modifiers– Measurement Procedures– Reported Results– Sample Data

• Motion

July 2006


Slide 4

doc.: IEEE 802.11-06/0965r1

Submission

Proposal Framework

• Usage case: Video streaming applications• Test Environments:

– Conducted Environment– OTA Environments

• Primary Metrics– Video Delivery

• Video Delivery Error Rate (VDER)

– Video Quality• Perceived Video Quality Mean Opinion Score (MOS)

• Secondary Metrics– Media Delivery Index (MDI)

• Media Loss Rate (MLR)• Delay Factor (DF)

July 2006


Slide 5

doc.: IEEE 802.11-06/0965r1

Submission

Test Output

• Video Delivery and Video Quality metrics are the primary output of the test

• MDI quantifies the impact of the network on the test outcome– Helps isolate root causes of poor performance

July 2006


Slide 6

doc.: IEEE 802.11-06/0965r1

Submission

Video Delivery and Video Quality

• There are two complementary ways to characterize video performance – video delivery and video quality

• Video delivery metrics quantify the impact of dropped, repeated or out-of-sequence video frames– Video delivery errors are caused by large-scale packet losses or

delay on the link

• Video quality metrics quantify the impact of video artifacts on end user perception– Video artifacts are caused by packet loss – Different codecs / content types translate levels of packet loss to

different levels of video quality

July 2006


Slide 7

doc.: IEEE 802.11-06/0965r1

Submission

Measuring Video Delivery and Video Quality

• Video Delivery Measurements– Video delivery measurements are made with video playback analysis

applications (VPAA)– VPAAs provide quantitative metrics that reflect the performance of the

underlying transport medium– A correlation can be established between video delivery measurements

and MOS of end user perception

• Video Quality Measurements– Video quality analysis applications (VQAA) output mean opinion scores

(MOS) that represent the expected end user perception of video quality– VQAAs are based on psychovisual models (such as VQM) which are

designed to characterize the quality of a perceived video clip relative to a source file

July 2006


Slide 8

doc.: IEEE 802.11-06/0965r1

Submission

The Media Delivery Index

• MDI defines media-weighted secondary metrics that quantify the impact of wireless link performance on observed video quality

• The media loss rate (MLR) reflects the portion of the video stream that was undelivered due to packet loss– The MLR is the packet loss rate expressed as a percentage of the video

stream– High MLR corresponds to video delivery errors (client buffer starvation)

or noticeable display artifacts from undelivered packets

• The delay factor (DF) quantifies the amount of buffering required to maintain an uninterrupted media stream– The DF is the required client buffer depth divided by the bitrate of the

video stream– High DF corresponds to non-fluid playback and loss of interactivity for

real-time applications such as video conferencing

July 2006


Slide 9

doc.: IEEE 802.11-06/0965r1

Submission

Video Delivery, Video Quality and Wireless Performance – Best Effort Traffic (i.e. UDP/RTP)

July 2006


Slide 10

doc.: IEEE 802.11-06/0965r1

Submission

Video Delivery, Video Quality and Wireless Performance – Guaranteed Traffic (i.e. TCP)

July 2006


Slide 11

doc.: IEEE 802.11-06/0965r1

Submission

References

IEEE 802.11-06/0964r1 “Video Performance”

IEEE 802.11-06/0651r1 “Video Delivery versus Attenuation in a Conducted Environment”

IEEE 802.11-06/0321r0 “Video Gross Error Detector Video over Wireless Methodology”

IEEE 802.11-06/0144r1 “Video over Wireless Methodology”

IEEE 802.11-05/1194r0, “Video Testing Methodology”

IEEE 802.11-05/0887r0, “Video Testing Strategy”

July 2006


Slide 12

doc.: IEEE 802.11-06/0965r1

Submission

Agenda


Test Description (doc.: 802.11-06/0964r1)– Introduction and Purpose– Resource Requirements– Test Setup– Video Content Preparation and Analysis– Calibration– Baseline Configuration– Modifiers– Measurement Procedures– Reported Results– Sample Data

• Motion

July 2006


Slide 13

doc.: IEEE 802.11-06/0965r1

Submission

Introduction and Purpose

• This test measures the video streaming performance of the link between a DUT and a WLCP– Video traffic is streamed to the DUT over the RF interface

– The DUT display data is captured and analyzed for video delivery errors and video quality defects

– Traffic measured by the 802.11 traffic analyzer is used to calculate the MDI (media loss rate and delay factor)

July 2006


Slide 14

doc.: IEEE 802.11-06/0965r1

Submission

Hardware Resource Requirements

• The basic test configuration is defined in section 5

• In addition, the following items are required:– A traffic generator capable of generating video traffic

• The video traffic generator can be a general purpose PC or dedicated device that streams video content to the DUT

– Video capture device, a PC or dedicated device capable of connecting to the DUT display port and saving video data to non-volatile storage

– The DUT may also act as the video capture device if it has the ability to save or process media streams in real-time (not progressive download)

July 2006


Slide 15

doc.: IEEE 802.11-06/0965r1

Submission

Software Resource Requirements

• A video playback analysis application capable of instrumenting video content for the detection of dropped, repeated and out-of-sequence video frames

• A video quality analysis application capable of analyzing video frames for image defects and rendering a quality score based on a perceptual model of the video format under test

• A video encoder application for encoding test media in the required format

• A video capture application for capturing display data from the DUT

July 2006


Slide 16

doc.: IEEE 802.11-06/0965r1

Submission

Test Setup

July 2006


Slide 17

doc.: IEEE 802.11-06/0965r1

Submission

Video Content Preparation

• Video content must be instrumented by the video playback analysis application before it can be used for this test

• Instrumentation ensures content-independent VDER results

• Uncompressed source content is first instrumented and then encoded in the desired test format before being loaded on the video traffic generator

• Preparation can be performed in real-time if supported by the video traffic generator

July 2006


Slide 18

doc.: IEEE 802.11-06/0965r1

Submission

Video Content Preparation Procedure

July 2006


Slide 19

doc.: IEEE 802.11-06/0965r1

Submission

Video Content Analysis

• After passing through the RF data connection, video content is captured from the DUT display and saved to a file (typically uncompressed)

• The instrumented video capture file is examined for video delivery error and video quality defects by the video playback analysis application and the video quality analysis application

• Analysis can be performed in real-time if supported by the video capture device

July 2006


Slide 20

doc.: IEEE 802.11-06/0965r1

Submission

Video Content Analysis Procedure

July 2006


Slide 21

doc.: IEEE 802.11-06/0965r1

Submission

Calibration

• Video performance tests require that the DUT support the video format, bitrate, network transport and streaming protocol used by the video traffic generator

• Video format compatibility must be verified before the test begins

• Video performance measurements can be affected by the capture device, so capture device performance must be verified with local video playback from the DUT

• If there are too many errors detected during calibration, the capture device should not be used for video performance tests

July 2006


Slide 22

doc.: IEEE 802.11-06/0965r1

Submission

Baseline Configuration

• Maximum transmit power setting

• QoS enabled with video traffic at high priority (no background traffic)

• No security

• No power management

• Video traffic configured to model desired application (i.e. 8 Mbps MPEG2 for DVD-quality video)

• Default DUT buffer size

• No content encryption

July 2006


Slide 23

doc.: IEEE 802.11-06/0965r1

Submission

Modifiers

• QoS disabled

• QoS enabled with background traffic at low priority

• MAC security

• Power management enabled, different profiles

• Alternate video bitrates and/or network transports

• DUT buffer size

• Video content encryption

July 2006


Slide 24

doc.: IEEE 802.11-06/0965r1

Submission

Measurement Procedure

• Load video content on the video traffic generator• Associate DUT with WLCP

Repeat the following steps for each attenuation or range increment:

• Start the video capture operation on the capture device and then start the video stream to the DUT

• After the specified test duration, stop the capture operation and then stop the video stream

• Analyze the resulting video capture file with the video playback analysis application and the video quality analysis application to measure the video delivery error rate and the video quality score

• Calculate MDI using metrics from the 802.11 traffic analyzer

July 2006


Slide 25

doc.: IEEE 802.11-06/0965r1

Submission

Reported Results

• The reported results should include the following items– Calibrated VDER, including the capture device baseline

– Video Quality MOS and DUT MOS upper bound

– MDI MLR and DF

– Detailed information about the test configuration including the video format, network transport and video capture device

Attenuation (dB) or Range (m)

Video Delivery Error Rate (Errors/minute)

Video Quality (MOS)

MDI – MLR MDI – DF

Sample Report:

July 2006


Slide 26

doc.: IEEE 802.11-06/0965r1

Submission

Sample Data

Compares video streaming performance of wireless adapters “A” and “B”

• Conducted environment

• Standard definition video (720x480 pixels)

• MPEG2 transport stream– 12 Mbps MPEG2 video

– Constant bitrate

– No audio

• UDP video transport

July 2006


Slide 27

doc.: IEEE 802.11-06/0965r1

Submission

Sample Data – Adapter A

July 2006


Slide 28

doc.: IEEE 802.11-06/0965r1

Submission

Sample Data – Adapter B

July 2006


Slide 29

doc.: IEEE 802.11-06/0965r1

Submission

Agenda


Test Description (doc.: 802.11-06/0964r1)– Introduction and Purpose– Resource Requirements– Test Setup– Video Content Preparation and Analysis– Calibration– Baseline Configuration– Modifiers– Measurement Procedures– Reported Results– Sample Data

Motion

July 2006


Slide 30

doc.: IEEE 802.11-06/0965r1

Submission

Motion

Move to instruct the editor to incorporate the contents of document 802.11-06/0964r1 into the P802.11.2 draft.

Technical (75%)

Y:

N:

A:

Documents

Doc.: IEEE 802.11-06/0965r1 Submission July 2006 Royce Fernald - Intel CorporationSlide 1 Video Performance Notice: This document has been prepared to