High Speed Data Networks Mobile Device Application Testing ... APEX UDP.pdf · High Speed Data Networks Mobile Device Application Testing Challenges ... HSDPA and HSUPA enabled devices

High Speed Data NetworksMobile Device

Application Testing Challenges

Ensuring Mobile Applications Success!

Spirent Wireless SeminarApril 2007

Proprietary - © 2006 Spirent Communications.

Agenda

Part I – Applications Enablers - Mobile Data Performance Analysis

Understanding how handset and device development can impact your upgrade to HSPAMethodology for analysing the performance of devices required to support HSDPA Identifying timescales for availability and widespread deployment of HSDPA and HSUPA enabled devicesDetermining realistic performance of devicesSpirent’s APEX UMTS Data Performance Module

Part II – Mobile Multimedia Application Performance


Need to answer questions such as, what is the …Actual time for data transfer?

Ability of the Device to accurately measure downlink channel quality and report CQI?

ApplicationApplication GoodputGoodput vs. Physical LayerPhysical Layer throughputthroughput rate?

Device performance under long service usage period?

Mechanisms to help benchmark UE data performance?

Statistical data needed to help establish “golden UE” performance?

Characterizing Data Performance


Why Test Goodput/Throughput Performance?

Layer 1 Layer 1 Throughput Throughput

Rates (Mbps)Rates (Mbps)

Air Air InterfaceInterfaceCDMA 2000

CDMA 2000UMTS

UMTS

Release 99

Release 99

HSDPA with

HSDPA with

QPSK only

QPSK only

HSDPA with

HSDPA with

16QAM16QAM

Mob

ile A

pplic

atio

nsM

obile

App

licat

ions

153 Kbps153 Kbps

2.42.4 3.13.1

384 Kbps384 Kbps1.81.8

14.414.4

SMSSMS MMSMMS

File File

SharingSharing

Image Image

SharingSharing

Short Video

Short Video

burstburst

Streaming

Streaming

VideoVideo

key enablerkey enabler of datadata--centric centric applications applications such as Mobile Videosuch as Mobile Video

major drivermajor driver for faster air interface technologies

no current benchmarksno current benchmarksto test Mobile Video throughput

GoodputGoodput/Throughput /Throughput ……

EVDO

EVDO

Release 0

Release 0EVDO

EVDO

Release A

Release A

Methodology for analysingthe performance of devices required to support HSDPA


Conformance testingConformance testing offers a starting point with some limitations

The realreal--world deploymentworld deployment scenarios require adapting to variablevariablenature of the networks, and the UE test conditions!

MeasurableNot Addressed

CriticalMulti-hour stress tests

Quantified Empirical Data

Pass/FailCriticalUE-to-UE comparative data


Critical“Safety margin”


CriticalL1, L7 throughput


CriticalThroughput sensitivity to key parameters

VariableFixedVariableTransport Block size

VariableFixedVariableMAC-hs parameters

Performance Testing

Conformance Testing

Real World Testing

Real-world conditions are constantly changing.

Conformance testing has limited value in assessing QoE.

Performance Testing simulates real-world test scenarios.

HSDPA Testing Challenges - Conformance Limitations


HSDPA is a highly dynamicdynamic and adaptiveadaptive technologyData rate variesvaries continuously as a function of the channel qualitychannel quality

Mobile devices constantly competecompete for a shared channelshared channel

Network has multiple parametersparameters for specific implementation

Data throughput volume is affected by protocol layer interdependenceinterdependence

ReliableReliable and repeatablerepeatable lablab--basedbased UE testing is needed to ensure necessary HSDPA test coverage prior to deployment …

HSDPA Testing Challenges – Variable Technology Characteristics

Identifying timescales for availability and widespread deployment of HSDPA and

HSUPA enabled devices


The Importance of Data Services

By all measures, data continues to grow as a percentage of totalOperator revenue

Timely deployment of HSPA networks will further fuel this growth

Source: GSA


HSPA Deployment: The Operators

121121 Operators deploying HSPDA in 5555 countries

63 commercial HSDPA networks in 38 countries have already launched

Deployment commitments are truly global:AmericasAmericas: 33 countries/territories

APACAPAC: 12 12 countries/territories

EuropeEurope: 3131 countries/territories

Middle East & AfricaMiddle East & Africa:99 countries/territories

Source: GSA GSM/3G Market/Technology Update, October 1, 2006Source: GSA GSM/3G Market/Technology Update, October 1, 2006


HSPA Deployment: The Mobile Devices

407 WCDMA/HSDPA devices launched by 44 suppliers

62 devices in the market (from 19 suppliers) that support HSDPA

Source: GSA GSM/3G Market/Technology Update, September 30, 2006Source: GSA GSM/3G Market/Technology Update, September 30, 2006


Deployment: Faster and Better, HSUPA

HSDPA 7.2 MbpsRX DiversityRX Diversity – Integral part of higher data rates

7.2 Mbps-capable devices already in the market

Firmware upgradeFirmware upgrade required to enable 7.2 Mbps

Commercial driver for network availability still unclearLate 2007? 2008?Late 2007? 2008?

HSUPAPre-commercial devices expected at beginning of 2007beginning of 2007First commercial data card launches in mid 2007mid 2007“Handheld” HSUPA devices begin to appear in 20082008

Determining Realistic Performance of Devices


Case Study - Multiple 3.6 Mbps (CAT 6) Devices under Static Conditions

Downlink rate of four HSDPA CAT 6 devices tested using APEX UDP under two sets of static channel conditions

Physical data throughput measurements shows expected results:Under favorable conditions all devices performed close to maximum rate (3.6 Mbps)

Under weakened conditions there is a drop in throughput rate ranging between (2.2 – 2.5) Mbps

Physical data throughput is not enough to provide a root-cause of poor performance.

Additional metrics such as Median CQI or the Mac-HS statisticsare helpful


Device (UE) physical layerhysical layer performance is consistent under strong RFstrong RF conditions and it begins to diverge as RF weakensRF weakens

under staticstatic conditionsconditions

Case Study - Applications Performance Differences under Static Conditions

Ask for a copy of our latest WhitepaperAsk for a copy of our latest Whitepaper……

APEX UDP resultsAPEX UDP results……

0

500

1000

1500

2000

2500

3000

3500

4000

HS-PDSCH = -3 dB / Ior = -60 dBm HS-PDSCH = -6 dB / Ior = -80 dBm

Static Channel Conditions

Configuration = HS-PDSCH = -3 or -6 dB, Ior = -60 or -80 dBm (No Noise)TTI = 1, # of H-ARQ = 6, CQI = Fixed at 22, TBS = Set per TR 25.214

Data

Thr

ough

put (

Kbps

)

UE AUE BUE C UE D

Throughput Differences @ weakened channel conditions(>10% when UE B is compared to UE D)

Reduction in Maximum Throughput @ weakened channel conditions (>30% drop on all CAT 6 UEs tested)


Case Study - Multiple 3.6 Mbps (CAT 6) Devices under Dynamic Conditions

Downlink rate of four CAT 6 devices tested using APEX UMTS Data Performance Module with four different fade models (PA3, PB3, VA30, PA3, PB3, VA30, VA120VA120)Performance of devices varied significantly under different fade modelsfade modelsand varying code powersvarying code powers

E.g. the throughput rate difference between UE D and other UEs is increased to 30%30% under PB3 & VA30PB3 & VA30 fading conditions

Variations observed between devices with the same chipsets, and between devices with different chipsets.

171344.05UE D

15932.5UE C

15930.51UE B

14878.68UE A

Median CQI

Throughput (kbps)

Device Under Test

UDP Downlink TestVA30 Fade Model with Ior/Ioc = 10 dB

Commercial devices are providingHSDPA capabilities, but their

performance differencescan be significant


Case Study - Multiple 3.6 Mbps (CAT 6) Devices under Static Conditions

End userEnd user’’s quality of experiences quality of experienceExample: UE B with 37.9% NAK at Ior = -80 dBm could lead to blank frames and/or blockiness in an application such as VT or Video Streaming

Accurate channel quality detection and reporting (CQI) is critical to:

Mac-HS Statistics (ACK / NACK / Stat DTX Performance) for CAT 6 UEs

Table 1. Mac-HS Statistics (ACK / NACK / Stat DTX Performance) for CAT 6 UEs

030.0369.97202507.98D

031.468.6212459.44C

037.9562.05212223.86B

032.9767.03202402.26A

Static Conditions – HS-PDSCH = -6 dB / Ior = -80 dBm (weakened conditions)

00100243583.99D

00100243583.99C

00.2599.75243575.08B

00.199.99243583.76A

Static Conditions – HS-PDSCH = -3 dB / Ior = -60 dBm (favorable conditions)

Stat DTX %NACK %ACK %Median CQI

Throughput

(Kbps)

UE Under Test


Case Study - Multiple 3.6 Mbps (CAT 6) Devices under Static Conditions Cont.

Overall network optimization and experience of all usersOverall network optimization and experience of all users

Accurate channel quality detection and reporting (CQI) is critical to:

Example: UE BUE B’’s s CQI compared to UE DUE D is asking for more resources while its higher NACK % NACK % indicates that it won’t be able to handle it. Under live conditions, this could mean that network is sending too much data to a user that might be in poor coverage resulting in high error rates, wasting shared resources.

Mac-HS Statistics (ACK / NACK / Stat DTX Performance) for CAT 6 UEs030.0369.97202507.98D

031.468.6212459.44C

037.9562.05212223.86B

032.9767.03202402.26A

Static Conditions – HS-PDSCH = -6 dB / Ior = -80 dBm (weakened conditions)

00100243583.99D

00100243583.99C

00.2599.75243575.08B

00.199.99243583.76A

Static Conditions – HS-PDSCH = -3 dB / Ior = -60 dBm (favorable conditions)

Stat DTX %NACK %ACK %Median CQI

Throughput(Kbps)

UE Under Test


Device (UE) Device (UE) application layerapplication layer performance varies significantly performance varies significantly under the under the dynamicdynamic fadingfading and and noisenoise conditionsconditions

Ask for a copy of our latest WhitepaperAsk for a copy of our latest Whitepaper……


Case Study - Applications Performance Differences under Dynamic Conditions

0

200

400

600

800

1000

1200

1400

1600

1800

2000

PA3 PB3 VA30 VA120

Fade Model (Per 34.121 - Appendix D)

Configuration = HS-PDSCH = -3 dB, Ior = -60 dBm, Ior/Ioc = 10 dBTTI = 1, # of H-ARQ = 6, CQI = Based on UE Reports, TBS = Set per TR 25.214

Dat

a Th

roug

hput

(Kbp

s)

UE AUE BUE C UE D

Clear Performance DifferencesClear Performance DifferencesUE D shows a 30% increase in data throughput rates when

compared to the other UEs tested.


Dynamic Dynamic conditions such as varying code powervarying code power levellevel directlydirectlyaffectsaffects the device application layer performance.

0

200

400

600

800

1000

1200

1400

1600

HS-PDSCH = -3 dB HS-PDSCH = -4 dB HS-PDSCH = -5 dB HS-PDSCH = -6 dB

Code Power Level (HS-PDSCH Level) - For Fade Model PB3 (Per 34.121 Appendix D)

Configuration = HS-PDSCH = -3 to - 6 dB, Ior = -80 dBm, Ior/Ioc = 10 dBTTI = 1, # of H-ARQ = 6, CQI = Based on UE Reports, TBS = Set per TR 25.214

Dat

a Th

roug

hput

(Kbp

s)

UE AUE BUE C UE D

Code Power Reduction yields Throughput Degradation1 dB = 16% (on average) & 2 dB = 29% (on average)

Case Study - Applications Performance Differences under Dynamic Conditions Cont.


Spirent’s APEX UMTS Data Performance Module


APEX Test Manager

Building Blocks

WCDMA/HSDPA CDMA 1X/EV-DO

Platform

Test ModulesUMTSData

Performance

Wireless Multimedia

Analysis

ReliableReliable and repeatablerepeatable lablab--basedbased device testing to ensure necessary HSDPA & R99 throughput performance prior to deployment …

Ensuring Mobile Applications Success with APEX UMTS Data Performance Module

ModularModular CustomizableCustomizable AutomatedAutomated


Answering Key Performance Questions

Packet Data RoutePacket Data Route

Fading

Multi-path

Delay

AWGN

UDP

TCP

FTP

User Defined

APEX UMTS DATA Performance ModuleAPEX UMTS DATA Performance Module

SpirentTest Manager

SR5500Channel Emulator

SR3420Network Emulator

RadioRadioAccessAccessNetworkNetwork

How well does application

layer goodputperform?

How well does application

layer goodputgoodputperform?

How does fading impact throughput/ goodput?

How does fading impact throughput/ goodputgoodput?

How well does physical layer

throughput perform?

How well does physical layerphysical layer

throughput perform?

For HSDPA & R99 DevicesFor HSDPA & R99 Devices……


Spirent Test Manager & UMTS Data Performance

Test Module

FTP ServerUDP Server

LAN

Test Client Laptop

SR3420BSS Simulator

RFRF

Architecture & Instruments

Controller PC

SR5500Fading Simulator

RXDiversity


The SR3420 WCDMA Network Emulator is …Built on the latest platform based on SpirentSpirent’’s global leadership in s global leadership in lablab--based wireless network emulationbased wireless network emulation

Designed specifically to address highhigh--speed dataspeed data

Upgradeable to track evolving industry requirements

Real-Time Network Emulation

SR3420 is built upon a real-time state machine

Resulting in a “Network Emulation much closer to a real network than others”

Provides greater HSDPA performance at all layers of the protocol stack


The SR5500 Wireless Channel Emulator is …Built on SpirentSpirent’’s bests best--inin--class channel class channel emulationemulation solution for real-life deployment scenarios …

RF impairments fully exercise, for example…Adaptive coding and modulation

HARQ with soft combining

Real-Time Channel Emulation

Allows integrated RX Diversity within the APEX UDP system

Supports all 3GPP fade models

Dynamic Environment Emulation (DEE)- provides real-network modeling capability


Spirent’s APEX Platform Expert Tools

NINI’’ssTestStandTestStandSoftware Software EngineEngine

SpirentSpirent’’s s Enhanced Enhanced Front EndFront End

Open & Open & CustomizableCustomizable

Easy & Easy & Comprehensive Comprehensive Test CreationTest Creation


Reduced ramp-up time

EasyEasy--toto--useuse, interactiveinteractive GUI requires minimal training

APEX UMTS Data Performance Module Offers Major Savings

Reduced test execution time

Repeatable, realRepeatable, real--worldworld conditions in a controlled lab environment

Reduced test implementation time

IntegratedIntegrated, turnturn--key solutionkey solution and powerful Test Manager environment allows rapid test implementation

Documents

High Speed Data Networks Mobile Device Application Testing ... APEX UDP.pdf · High Speed Data Networks Mobile Device Application Testing Challenges ... HSDPA and HSUPA enabled devices