Doc.: IEEE 802.11-06/0321r0 Submission March 2006 Royce Fernald - Intel CorporationSlide 1 Video Gross Error Detector Video over Wireless Methodology Notice:

March 2006

Royce Fernald - Intel Corporation

Slide 1

doc.: IEEE 802.11-06/0321r0

Submission

Video Gross Error Detector Video over Wireless Methodology

Notice: This document has been prepared to assist IEEE 802.11. It is offered as a basis for discussion and is not binding on the contributing individual(s) or organization(s). The material in this document is subject to change in form and content after further study. The contributor(s) reserve(s) the right to add, amend or withdraw material contained herein.

Release: The contributor grants a free, irrevocable license to the IEEE to incorporate material contained in this contribution, and any modifications thereof, in the creation of an IEEE Standards publication; to copyright in the IEEE’s name any IEEE Standards publication even though it may include portions of this contribution; and at the IEEE’s sole discretion to permit others to reproduce in whole or in part the resulting IEEE Standards publication. The contributor also acknowledges and accepts that this contribution may be made public by IEEE 802.11.

Patent Policy and Procedures: The contributor is familiar with the IEEE 802 Patent Policy and Procedures <http:// ieee802.org/guides/bylaws/sb-bylaws.pdf>, including the statement "IEEE standards may include the known use of patent(s), including patent applications, provided the IEEE receives assurance from the patent holder or applicant with respect to patents essential for compliance with both mandatory and optional portions of the standard." Early disclosure to the Working Group of patent information that might be relevant to the standard is essential to reduce the possibility for delays in the development process and increase the likelihood that the draft publication will be approved for publication. Please notify the Chair <[email protected]> as early as possible, in written or electronic form, if patented technology (or technology under patent application) might be incorporated into a draft standard being developed within the IEEE 802.11 Working Group. If you have questions, contact the IEEE Patent Committee Administrator at <[email protected]>.

Name Company Address Phone Email Royce Fernald Intel 5200 Elam Young Pkwy

MS:HF3-96 Hillsboro, OR 97124

503-696-4318 [email protected]

Connie Wang Intel 5200 Elam Young Pkwy MS:HF3-96 Hillsboro, OR 97124


Audrey Younkin Intel 5200 Elam Young Pkwy MS:HF3-96 Hillsboro, OR 97124


Uriel Lemberger

Intel Matam Industrial Park PO Box 1659 Haifa 31015 Israel

+972-4-865-5701

[email protected]

Authors:

March 2006


Slide 2

doc.: IEEE 802.11-06/0321r0

Submission

Abstract

• Goals• Video Gross Error Detector Overview• Base Test Configuration• Conducted Test Configuration• System Requirements• Calibration• Test Procedure• Modifiers• Sample Report• Caveats

March 2006


Slide 3

doc.: IEEE 802.11-06/0321r0

Submission

Goals

• Tie GED subjective assessment results into objective performance metrics – measure acceptable user experience threshold for video streaming over devices under test

• Use conducted configuration to demonstrate video over wireless testing with the GED

March 2006


Slide 4

doc.: IEEE 802.11-06/0321r0

Submission

Video GED Overview

• The Video Gross Error Detector is a high-level video performance analysis application

• Provides objective data on dropped, repeated or out-of-sequence frames

• Enables automated, quantitative, repeatable measurements of playback smoothness and frame rate stability

• Other video quality analysis tools (such as VQM) are used in addition to the GED to measure image quality

March 2006


Slide 5

doc.: IEEE 802.11-06/0321r0

Submission

Platform Independence

• The GED supports any video playback solution that can be connected to a hardware capture device

• Since the GED operates directly on capture files, it can measure and compare the video performance of various network transports, operating systems, streaming applications, media players and compression formats

• The GED allows direct comparison between PC platforms and consumer electronics devices

March 2006


Slide 6

doc.: IEEE 802.11-06/0321r0

Submission

GED Methodology• Source clips are encoded with GED frame identifiers, a

sequence of color blocks

• Video clips are played through the system under test and captured

• The GED reads the color blocks to locate dropped or repeated frames

GED Encode

March 2006


Slide 7

doc.: IEEE 802.11-06/0321r0

Submission

GED Processing0. Source Material 1. Marked Source Material

2. Compressed and Marked

4. GED Analysis

GED Encode Video Encoder

System Under Test

3. Capture Results

GED Decode

March 2006


Slide 8

doc.: IEEE 802.11-06/0321r0

Submission

Base Equipment Requirements

• Test controller

• Media server

• Ethernet switch

• Wireless AP

• Client system – desktop, laptop or CE device with video over wireless support

• Video capture system

• Display device

March 2006


Slide 9

doc.: IEEE 802.11-06/0321r0

Submission

Base Test Configuration

March 2006


Slide 10

doc.: IEEE 802.11-06/0321r0

Submission

Conducted Equipment Requirements

• Base configuration

• Variable attenuator

• RF shield boxes

March 2006


Slide 11

doc.: IEEE 802.11-06/0321r0

Submission

Conducted Test Configuration

March 2006


Slide 12

doc.: IEEE 802.11-06/0321r0

Submission

Media Server Requirements

• Typical desktop PC, laptop or capable PDA

• Video streaming software – TCP, UDP or RTP transports

March 2006


Slide 13

doc.: IEEE 802.11-06/0321r0

Submission

Video Capture System Requirements

• Video capture adapter that supports the format of the display device (s-video, HDTV component, DVI, VGA, etc.)

• High-performance disk array for real-time capture of uncompressed video

March 2006


Slide 14

doc.: IEEE 802.11-06/0321r0

Submission

Client System Requirements

• Desktop, laptop, PDA or CE device with video over wireless support

• Media player application that supports the streaming format of the media server

March 2006


Slide 15

doc.: IEEE 802.11-06/0321r0

Submission

Media Preparation

• Encode each source clip with GED identifiers

• If testing a sequence of short clips, concatenate them with a video editor after the GED encode – this method is recommended because it allows testing with a wide range of video types

• Once encoded and assembled, compress the video clip in the desired format, such as MPEG-2

• Copy the clips to the media server and the system under test (for local playback calibration)

March 2006


Slide 16

doc.: IEEE 802.11-06/0321r0

Submission

Test Calibration

• Video capture hardware is not perfect and will sometimes introduce additional dropped or repeated video frames

• To measure the error introduced by the capture process, play the test video sequences locally on the DUT while capturing the results

• Analyze the capture file with the GED to determine the baseline error level

• Several hours of calibration should be performed for an accurate assessment of capture device errors

March 2006


Slide 17

doc.: IEEE 802.11-06/0321r0

Submission

Test Parameters

• Video clip duration – 5 minutes

• GED pass/fail score – 30 errors per clip (Mean Opinion Score > 4.8)

March 2006


Slide 18

doc.: IEEE 802.11-06/0321r0

Submission

Test Procedure

• Starting at the optimal value, increase attenuation in increments of 5 dB

• At each attenuation level, start video playback from the client device

• Once the client finishes initial buffering, start capturing video

• After the clip has finished streaming, stop the capture operation and analyze the capture output with the GED

• If the clip passes, increase attenuation by 5 dB

• If the clip fails, return to the last passing attenuation and increase in increments of 1 dB (record at least ten 1 dB steps above the breaking point)

• Conduct throughput tests for the ten attenuator settings

March 2006


Slide 19

doc.: IEEE 802.11-06/0321r0

Submission

Test Modifiers

• Video resolution

• Compression format

• Bitrate

• Number and type of video and audio streams

• Network transport type and settings

• Server and client software

• AP configuration

• Client device configuration

• Wireless channel

March 2006


Slide 20

doc.: IEEE 802.11-06/0321r0

Submission

Conducted Test Report

The following fields should be included in the test report

• Attenuator setting

• Average throughput

• GED score

• Pass/fail results (pass = fewer than baseline plus capture system calibration)

March 2006


Slide 21

doc.: IEEE 802.11-06/0321r0

Submission

Sample ReportSD Video (480i59.94, 720x480)

MPEG-2 10 Mbps CBR

AC-3 audio 256 Kbps

Media Server X, TCP streaming

AP X

Attenuation (dB) Throughput (Mbps) GED Score Result

86 15.1 2 Pass

87 14.3 3 Pass

88 14.4 2 Pass

89 12.0 5 Pass

90 11.9 5 Pass

91 11.7 7 Pass

92 10.5 12 Pass

93 9.9 212 Fail

Client platform X, Wireless card Y, Media player Z

g-only mode, channel 6

Capture device calibration: 2 errors

Pass/Fail criterion: 32 errors

March 2006


Slide 22

doc.: IEEE 802.11-06/0321r0

Submission

Caveats

• OTA video tests are less deterministic than conducted measurements

• Statistical methods must be used to characterize expected video performance in practical OTA environments

March 2006


Slide 23

doc.: IEEE 802.11-06/0321r0

Submission

References

IEEE 802.11-06/0144r1 “Video over Wireless Methodology"

Documents

Doc.: IEEE 802.11-06/0321r0 Submission March 2006 Royce Fernald - Intel CorporationSlide 1 Video Gross Error Detector Video over Wireless Methodology Notice: