Network Assurance and Testing During the Migration to VoIP

Analyze Assure Accelerate

TMC Developers ConferenceSan FranciscoAug 03rd, 2005

Andy Huckridge

Spirent Communications.

Chair, Interop WG, MSF

ST-09 Network Assurance and Testing During the Migration to VoIP


Agenda

• Spirent overview

• Key implementation issues

• What is Triple Play / Converged networks?

• Specifics on testing SIP

• Network Impairments and Parameters that Voice and Video Affect Quality

• Metrics for Measuring Voice and Video Quality and Performance

• Good test methodology


Spirent Communications

• Spirent is the test solution leader

– 1,800 employees in 14 countries

– More than 1,500 customers

– Sales and service capabilities in 30 countries


PerformancePerformanceAnalysisAnalysis

Industry standards

PerformanceTesting Functionality &

Conformance Testing

ManufacturingQuality Assurance

ServicesServicesDeploymentDeployment

• Characterize system before trial

• Validate system scalability

• Identify capacity limits

• Measure call performance

• Automate regression testing

• Characterize system before trial

• Validate system scalability

• Identify capacity limits

• Measure call performance

• Automate regression testing

Implementation steps - Lab


ServiceServiceAssuranceAssurance

Pilot Networks

Network Certification

Initial Deployment

WidespreadNetwork Deployment

• Facilitate vendor selection

• Identify performance ceilings

• Enable accurate capacity planning

• End-to-end service assurance testing

• Improve operational performance

• Improve customer satisfaction

• Facilitate vendor selection

• Identify performance ceilings

• Enable accurate capacity planning

• End-to-end service assurance testing

• Improve operational performance

• Improve customer satisfaction

Implementation steps - Network


Key Implementation Issues

• Circuit to packet migration

• Scalability and Performance

• Voice quality

• Interoperability and conformance

• Budget pressures


Before you deploy!

• Network Equipment Manufacturers (Chips, IP-PBX, Gateways, MSs & SSs)

– Characterize your system before trial

– Validate system scalability

– Identify capacity limits

– Measure call performance

• Service Providers(NSPs, SPs, ITSPs)

– Define criteria for vendor selection

– Identify performance ceilings

– Accurately plan for your capacity needs

– End-to-end service assurance testing


Data Transmission“Non-Real-Time” Applications

Name ResolvingDNS

File TransferFTP

Data BaseMS SQLOracle

WebHTTP

Email and MessagingPOPSMPTExchange

Telenet

Data Examples: Internet access, Email, File Transfer, Portals, Database Applications, Gaming, Government Services, Online Commerce

Music DownloadingHome control


Voice and Video “Real-Time Applications”

VoIP, IP Telephony, Video TelephonyG.711, G.729, G.728, G.726, G.723

H.261, H.263, SIP, SIP-T, H323, Skinny, MGCP,

MEGACO/H.248

VoDIP Music/Audio/Radio

IPTV ServicesBroadcast, On-Demand, Bi-directional / InteractiveMPEG1, MPEG2, MPEG4, VC1, H264

Multi-MediaRTP

H.264,Microsoft AVI, QuickTime (.mov)Windows Media (.wmv, .asf), RealMedia (.rm),

Voice Applications: Phone service integrated with videoVideo Applications: Broadcast TV, video on demand, distance learning

Real-Time Online CommunicationsInstant MessengerWebexNetmeetingSIPH.323

GamingSingle / Multiplayer


Off HookDial #

Dialed Digits

ConfigureConfigure

ConnectConnect

Notify

Ring Off HookHello

Hello

Voice Conversation

Good Bye

Good ByeOn Hook

On Hook

Disconnect

Configure

Signaling Path

Ring Back

Converged Triple Play: Data, Voice and Video With Network Impairments

Data

Video

Impairments can be heard in the voice conversation


Testing SIP Conformance

• Comprehensive and scriptable SIP call flows

• Complete configurable SIP signaling messages

• SIP protocol analysis

• Simplified flow diagrams with visual analysis

• Comprehensive conformance test suites


Testing SIP Conformance

• ETSI TS 102-027-1 v2.12,Tiphon:

– RFC 3261 user agent, proxy and redirect server compliancecompliance

• Graphical SDL and TTCN tools

– Create, edit, compile and execute simulation scripts and conformanceconformance tests

• Additional SIP messages beyond RFC 3261

– Included in torture teststorture tests

• Additional tests as defined by the SIP Forum


Testing with Configurable SIP

Configurable SIPConfigurable SIP call setup and call teardown

• Configurable call flows and messagesConfigurable call flows and messages

• Incoming message filterIncoming message filter

–Adaptive signaling syntax for SIP

– Improves interoperabilityinteroperability with new drafts and non-conformant proprietary implementations


Testing with Configurable SIP

• Configurable messagesConfigurable messages:

Invite, ACK, bye, register

Responses: 1xx, 2xx, 3xx, 4xx, 5xx, 6xx

• Configurable timers,Configurable timers, message intervalsmessage intervals

• Enable and disable optional messages: Re-invite, cancel, options, message, info, notify,

subscribe, unsubscribe, update, refer, Prack

Fix erroneous incoming messages “on the fly” with the “search and replace” method

Allows interoperability with SIP devices (including drafts, non conformant, prototype)


SIP Message Registration Screen Shots


SIP Message Origination


SIP Message Termination


Incoming Message Filters


TOS for SIP Signaling


Diffserv for SIP Signaling


Testing SIP Robustness

• Robustness testingRobustness testing

– Passed: does not crash, stable, or acceptable results

– Failed: crashes, unstable, or unacceptable results

• Security testingSecurity testing

– It is crucial to identify SIP security holesIt is crucial to identify SIP security holes

• SIP testing toolSIP testing tool

– Tests SIP robustness and security

– Comprehensive negative test suites for SIP


Real Signaling with real RTP

• Capability to do signaling with audio

• Capability to perform real time measurements

• Capability of using signaling without audio

• Problems of not using real signaling

• Problems of not using real RTP streams

• Real time objective metrics


Testing SIP-T

PSTN

SIP-T

SIP-TSIP-T

SS7 SS7

SIP ProxyH.248/Megaco H.248/Megaco

Trunking Gateway

Trunking Gateway

RTP/RTCP

POTS

MGC

SS7GR303ISDNCASV5

1000Base-SX/LX10/100/1000Base-T

PSTN

VoIP Network

POTS

MGC


SIP-T Performance Testing Suites

Performance testing

– Validate and stress-test SS7 ISUP and SIP interworking with optional media, over thousands of emulated user agents

• SIT-T testing

– Configurable SIP-T calls with intelligent protocols• QoS and CoS testing

– Optional TOS/Diffserv and VLAN options in SIP-T media calls, used to measure QoS with PESQ and e-model

• Feature testing

– Automated and configurable SIP call set-up, teardown, flows, messages


Network Impairments and Parameters that Affect Voice and Video Quality

• Network Architecture

• Types of Access Links

• QoS controlled Edge Routing

• MTU Size

• Packet Loss (Frame Loss)

• Out of order packets

• One Way Delay (Latency)

• Variable Delays (Jitter)

• Background Traffic (Congestion, Bandwidth, Utilization, Network Load, Load Sharing)

• Timing Drift

• Route Flapping

• Signaling protocol mismatches

• Network faults

• Link Failures

• Voice Only Impairments– Echo

– Voice coding algorithms

– A/D and D/A Conversion

– Noise – Circuit and External

• Video Only Impairments– Video coding algorithms

– Fixed vs Variable Frame Rate


IP Network Architecture

Core IP NetworkLAN A

Local AccessB

1000BaseX* 100BaseT Switch100BaseT Hub10BaseT* WLAN (~4 Mbit/s)----------------------Occupancy levelPacket loss

64 kbit/s*128 kbit/s256 kbit/s*384 kbit/s512 kbit/s*768 kbit/s

*T1 (1.536 kbit/s)E1 (1.920 kbit/s)E3 (34 Mbit/s)*T3 (44 Mbit/s)

ADSL (~256 kbit/s)*Cable (~256 kbit/s)Fiber (1-10 Gbit/s)

--------------------Occupancy levelQoS edge router

LAN B

Route flappingOne-way delay

JitterPacket loss

DestinationDevice B

Local AccessA

64 kbit/s*128 kbit/s256 kbit/s*384 kbit/s512 kbit/s*768 kbit/s

*T1 (1.536 kbit/s)E1 (1.920 kbit/s)E3 (34 Mbit/s)*T3 (44 Mbit/s)

ADSL (~2 Mbit/s)*Cable (~3 Mbit/s)Fiber (1-10 Gbit/s)

--------------------Occupancy levelQoS edge router

SourceDevice A

* Case used in impairment tables

1000BaseX* 100BaseT Switch100BaseT Hub10BaseT* WLAN (~4 Mbit/s)----------------------Occupancy levelPacket loss

Affects Data, Voice and Video Quality


Network Operating With Constant Delay

Affects Voice and Video Quality


End to End Delay Sources

• Algorithmic delay

• Serialization delay

• Propagation delay

• Component delay


Fixed• Look ahead

• Encoding

• Buffer

• VAD

• Packetizing

Fixed• Switching

Variable• Voice contention

• Data Contention

• Video Contention

Fixed• Serialization

WAN

Fixed• Switching

•Propagation

•Serialization


• Data Contention

•Video Contention

Fixed• Switching


• Data Contention

• Video Contention

Fixed• Decoding

Variable • De-jitter buffer

• Packet loss Concealment

Originating LAN

Core Network Terminating

LANOriginating Gateway

Edge Router Core

Network Routers

Terminating Gateway

Edge Router

Fixed• Serialization

WAN


Echo Impairment on Converged network

VV

T1 Link

Echo Canceller in MG reduces Echo Level

IP Network

ERLERLE

TELR

PBX POTSPhone

ERLE – Echo Return Loss EnhancementERL – Echo Return LossTELR – Talker Echo Loudness Rating

2 Wire

PBX

Analog 4-Wire Link

E&M

POTSPhone

Hybrid Transformer

Analog 2-Wire Link

TX

RX

TX

RX

T1 Link

Impedance Mismatch

MG

Tail CircuitIP

Phone

IP Phone

Delay in IP Network makes Echo sound worse

Affects Voice Quality


Echo Impairment on Converged network

Converged Network

Echo Path Side A (250ms)Echo Path Side B (250ms)

Path A to BPath B to A

Echo is caused by impedance mismatches in hybrid circuits (2w to 4w) and feedback between the telephone mouth piece and ear piece

Electrical Coupling• Impedance Mismatch (Hybrid)

Acoustical Coupling• Speakerphone



Effect of Delay on Voice Quality

PSTN

> 25ms Echo Cancellation Required

<150 ms (with echo cancellation): acceptable

> 400 ms unacceptable for most applications

150-400 ms: acceptable if delay expected

Voice Quality


Effect of Echo Level on Voice Quality

TELR – Talker Echo Loudness Rating(Signal to Echo Ratio)

More Echo

Less Echo

Affects

Voice Quality

Less Echo

More Echo


Network with Variable Delays (Jitter)

• Variable processing delay

– A busy router or switch will take longer to look up the routing (address) table

• Queuing delay

– Network congestion

Time (s)

Del

ay (

ms)

Affects

Voice and Video Quality


Jitter Characteristics

Time (s)

Del

ay (

ms)

Del

ay (

ms)

Del

ay (

ms)

Good

Bad

Severe



Packet LossExample: Queue Management

Bit Bucket

Threshold

RED (Random Early Discard)



Speech Compression Impairment

G.711 Best Quality

Common Compression Types: G.711, G.729, G.728, G.726, G.723, AMR, EVRC

Voice Quality


VAD – Voice Activity Detection

No VAD

VAD

Data is intentionally not sent during times of Silence


Timing may be different


= 80 ms Speech= 80 ms Speech



80 Bytes80 Bytes

40 Bytes40 Bytes

20 Bytes20 Bytes

Impact Of Packet Size

• Typically Packets are kept small for best results

• Many equipment manufacturers use dynamic packet size to optimize for network conditions

Normal size for VoIP applications


10 Bytes10 Bytes



Mechanisms for Assuring QOS

• Class of Service (COS) ITU-T Y.1541 defines the 5 classes of service and their application

• Type of Services (TOS)

• TOS and COS are both elements with in an IP Packet

• DIFSER and RSVP provide mechanisms to improve QOS

QoS Class(Y.1541)

Applications (Examples) Node Mechanisms Network Techniques

0Well

Managed

Real-Time, loss sensitive, Jitter sensitive, high interaction (VoIP, VTC, IPTV

Strict QoS. Guaranteed no over subscription on links.

Constrained Routing and Distance

1Best

Effort

Real-Time, Jitter sensitive, interactive (VoIP, VTC).

Separate Queue with preferential servicing, Traffic grooming

Less constrained Routing and Distances

2 Transaction Data, Highly Interactive, (Signaling) Separate Queue, Drop

priority

Constrained Routing and Distance

3 Transaction Data, Interactive

Less constrained Routing and Distances

4 Low Loss Only (Short Transactions, Bulk Data, Video Streaming)

Long Queue, Drop priority Any route/path

5Internet

Traditional Applications of Default IP Networks

Separate Queue (lowest priority)

Any route/path

Video

Data

Voice

Triple Play



TIA-921 and ITU-T G.NIMMTest Profiles Based on QoS (Y.1541) Classes

Impairment Type Units Range

Jitter ms 0 to +/- 250

One Way Latency ms 50 to 400

Sequential Packet Loss #sequential packets

2 to 500

Rate of Sequential Loss sec-1 < 10-1

Random Packet Loss % 0 to 20

Out of Sequence Packets % 0 to 20


Jitter ms +/- 75



2 to 5

Rate of Sequential Loss sec-1 0 to 2

Random Packet Loss % 0 to 2

Out of Sequence Packets % 0 to 0.1

Profile C Un-Managed Network

Table 4

Profile BBest Effort Managed Network

Table 3

Profile AWell Managed Network

Table 2

Different test profiles for different Service Level Agreements (SLAs)


Jitter ms +/- 50



Random loss only

Rate of Sequential Loss sec-1

Random Packet Loss % 0 to 0.05

Out of Sequence Packets % 0 to 0.001


Early Voice Quality Testing


Voice Quality Testing

• Active (Intrusive) Testing– Sends, Receives and compares Wave Files to measure voice quality

– MOS (Mean Opinion Score)

– PSQM, PSQM+ (Perceptual Speech Quality Measurement)

– PESQ (Perceptual Evaluation of Speech Quality)

– R-Value and J-MOS derived from PESQ

• Passive Testing– R-Value – ITU-T P.VTQ

• Measures Voice Quality on RTP Packets

• Based on E-model

• Japan – J-MOS

• Similar Techniques can be used to measure Video Quality

– P.563 (ITU-T recommendation) 3SQM, P-Stream• Measures Voice Quality of Voice traffic based on Audio Siginal

• Provides an estimate of PSQM


Active (Intrusive) Voice Quality Testing

DUTSend Wave Files

Example: (ITU-T Female Nice File with Pilot Tone)

Receive Wave Files

Measures Voice Quality by Comparing Sent and Received Wave files

MOS, PSQM, PSQM+, PESQ, R-Factor (PESQ Derived)

Sent (Green) and Received (Orange) wave files

Expanded Sent (Green) and Received (Orange) wave files

PESQ Score vs Number of PESQ Measurements

Values are different for Male, Female, different Wave Files and different Languages


Passive Voice and Video Quality Testing

R-Factor/Emodel

TrunkingGateway

E1/T1/E3/T3/PRI/GR303,

V5,SLC96

RTPPSTNPSTN

IP Network

IP Telephone

Measure:Video QualityMOS-LQMOS-CQMOS-PQJ-MOSNetwork RUser RBurst statisticsDiagnostic data

Measure: Video QualityMOS-LQMOS-CQMOS-PQJ-MOSNetwork RUser RBurst statisticsDiagnostic data

RTP

ITU-T P.VTQ


Passive Voice Quality TestingP.563 (P-Stream, 3SQM)

DUT Receive Audio

Estimates Voice Quality based on 3 Characteristic of Received Audio

RTP


Voice Quality Measurements

P.861PSQM/PSQM+

0

6.5

PAMS

1

5

Emodel P.862PESQ

~3.88

~3.65

~3.40

~3.13

~2.84

-0.5

4.5


Sample Voice Quality Test Results

G.711 G.723.1 (6300 bps)

Comparison of Scores for G.711 and G.723.1 (6300 bps)

PSQM

PESQ

MOS

Created by inducing packets lost


• Video Compression– Video Compress schemes affect the video quality

– H.261, H.263, H.264, VC1, MPEG-1, MPEG-2, MPEG-4, Microsoft AVI, Windows Media (.wmv, .asf), RealMedia (.rm), QuickTime (.mov)

• Interactive real-time applications (e.g., video conferencing, voice over IP) are sensitive to latency and Frame Rate

• Typical Video Quality Metrics– Objective MOS

– Blockiness

– Blur

– PSNR

– Spatial Resolution

– Temporal Resolution

Video Quality Measurement

– SNR

– Edge Noise

– Jerkiness

– Error Blocks

– Object Retention

– Color Reproduction Accuracy


Video Quality MeasurementBlockiness

OriginalBlockiness


Video Quality MeasurementReference and Blocky Video

Original(Reference)

Blocky


Video Quality MeasurementBlur

OriginalBlur


Video Quality MeasurementNoise

OriginalNoise


Video Quality MeasurementSpatial (Pixel) Resolution

128X128 32X32 8X8

Spatial Resolution

Department of Computer ScienceUniversity of Canterbury http://www.cosc.canterbury.ac.nz/people/mukundan/covn/Imgresl.htm


Video Quality Measurement Temporal (Motion) Resolution

Video Frames

Fk Fk+1 Fk+2 Fk+3 Fk+4 Fk+5

Temporal-Width (t)

Vertical-Width (v) Horizontal-Width (h)


Video Quality MeasurementModels

• Video Quality Metrics (VQM)– ITU-T SG9 and VQEG are working on standard

• TV Model - optimized for higher bit-rate digital television systems with no frame dropping (e.g., MPEG-2)

• Videoconferencing Model - optimized for lower bit-rate videoconferencing systems that drop frames (e.g., H.261, H.263).

• General Model - optimized for a wide range of video quality (videoconferencing, TV)

• Developer Model - optimized for a wide range of video quality (videoconferencing, TV) with the added constraint of fast computation.

• PSNR Model - based on the traditional peak signal-to-noise-ratio (PSNR) calculation.


Video Quality MeasurementTechniques

• Full Reference (ITU-T J.144R and BT.1683)– Video quality is calculated by comparing the received video with

the complete original video

• Reduced Reference (ANSI T1.801.03-2003 and ITU-T J.143) – Spatial and temporal information are calculated from original

video and transmitted to the receiving end

– Video quality is calculated by comparing the received video with the reduced reference

• No Reference – Passive – Video quality on based only on the received information (picture

content)

– Video quality is derived from RTP packet information similar to R-Factor (E-Model) for voice quality


Video Quality Measurement Full Reference

Full Reference

Video Quality Score

•MOS

•Blockiness

• Blur

• PSNR

Processor Processor

Original Video Received Video

Original Video

Transmitted Video Signal


Video Quality Measurement Reduced Reference

Reduced-Reference •Spatial (Pixel) and Temporal (Motion) Information

Reduced-Reference SignalLow Rate

Processor Processor

Transmitted Video Received Video

Video Quality Score

•MOS

•Blockiness

• Blur

• PSNR


Video Quality Measurement No-Reference

No-ReferencePicture Content

or Passive monitoring of RTP

Processor Processor

Transmitted Video Received Video

Video Quality Score

•MOS

•Blockiness

• Blur

• PSNR


Types of TestingType of tests

• Video Quality

• Voice quality

• Functional

• Scalability

• Troubleshooting

• Conformance

• Interoperability

• Triple Play (Data, Voice and Video)

Payload types

• Video

• Voice

• Data

• Fax

• Modem

With these protocols

• IP

– SIP

– H.323

– MGCP

– Megaco/H.248

– Skinny

• PSTN

– CAS

– PRI

– SS7

– NFAS

– V5

– GR303

With these interfaces

• GigE 1000Base-SX, 1000Base-LX

• 10/100/1000Base-T

• Analog

• T1/E1

• T3/E3

Test these DUTs

• IP PBX

• Gateways

• IP Phone

• Servers

• Firewalls

• IAD


Types of Testing

•Call Establishment

–Start Dial Signal Delay

–Post Dial Delay

–Call Duration

–Ring Duration

•Call Disconnect

–Connection Disconnect Delay

–Release on Request

•Call Statistics

–Connection set-up failures

–Connection premature disconnect

–Call completion percentages

•Speech Quality Measurements

–PESQ

–PSQM

–MOS

–R Factor

–Echo Delay

–Round Trip Delay

–Echo Return Loss

–Signal Pass Noise

–Noise Level

•Video Quality Measurements

–MOS

–Blockiness

– Blur

– PSNR

•Transport Layer Measurements

–One-way Transmission Time

–Roundtrip Transmission Time

–Jitter

–Packets out of order

–Packet Loss


Call Quality Summary

Fair24%

Poor2% Bad

1%

Good 73%

Good

Fair

Poor

Bad

Call Quality Summary

Not Recommended1%

Nearly All Users Dissatisfied

2%Many Users Dissatisfied

10% Very Satisfied20%

Some Users Dissatisfied

14%

Satisfied53%

Video Telephony TestingDistributed Testing

75

80

85

90

95

100

% C

om

ple

te

% Complete

Call Completion Rate by Day

% Complete 98 97 99 99 96 99 99

Mon Tue Wed Thu Fri Sat Sun

0

20

40

60

80

100

120

140

160

180D

ela

y (

ms)

One Way Delay by Call Group

Delay (ms) 122 98 106 173 132 112

NYC - DAL NYC - CHI NYC - SJ DAL - CHI DAL - SJ CHI - SJ

0.00

0.20

0.40

0.60

0.80

1.00

1.20

1.40%

Lo

ss

Packet Loss by Call Group

% Loss 0.10 0.43 0.40 1.23 0.35 0.64


0

50

100

150

200

250

300

350

Cal

l S

etu

p T

ime

(ms)

Call Setup Time by Call Group

Call Setup Time (ms) 150 175 130 313 110 105


0

10

20

30

40

50

60

70

80

Jitt

er (

ms)

Jitter by Call Group

Jitter (ms) 43 51 41 73 54 45


0

20

40

60

80

100

Jitt

er (

ms)

Jitter (ms)

Jitter by HourCHI-to-DAL

Jitter (ms) 77 69 67 65 68 68 69 70 73 75 76 93 92 100 82 83 81 80 79 79 79 79 79 78

12 AM

1 AM

2 AM

3 AM

4 AM

5 AM

6 AM

7 AM

8 AM

9 AM

10 AM

11 AM

12 PM

1 PM

2 PM

3 PM

4 PM

5 PM

6 PM

7 PM

8 PM

9 PM

10 PM

11 PM

1.00

2.00

3.00

4.00

5.00

MO

S

MOS

Call Quality Summary by HourCHI-to-DAL

MOS 3.5 3.7 3.6 3.4 3.8 3.4 3.6 3.8 3.5 2.9 2.8 2.3 1.9 1.8 2.9 2.9 3.1 2.9 3.4 3.5 3.5 3.6 3.6 3.7

12 AM

1 AM

2 AM

3 AM

4 AM

5 AM

6 AM

7 AM

8 AM

9 AM

10 AM

11 AM

12 PM

1 PM

2 PM

3 PM

4 PM

5 PM

6 PM

7 PM

8 PM

9 PM

10 PM

11 PM

• Isolate Network Problems

• Results Over Time

• Results by Group


Good test methodology

Implementation, Validation & Observation– Conformance testing

• IETF 3261 & new SIP RFC’s

– Stress testing• Scriptable call flow

• Bulk signaling with real RTP

– Robustness testing• SIPPING Torture Test

• PROTOS / ETSI TIPHON

– Visual protocol analysis• Application & content decoding



Documents

Network Assurance and Testing During the Migration to VoIP