Wi-Fi performance measurements · § What share of packets use ... – How much airtime captured ... Performance metrics, active/passive test hp:

Copyright © 2015 7signal Solutions, Inc.

RoadmapJan 11th , 2016

Wi-Fi performance measurements

Feb 23rd, 2016Veli-Pekka Ketonen

CTO, 7signal Solutions

Material atgo.7signal.com/WLPC2016

@VPonwireless


Data collection methods

Ac#vetestmetrics

Beaconanalysismetrics

Packetanalysismetrics

Networkelement

countermetrics

Fullpacketcapture

Spectrumanalysismetrics

2


Data collection tools

3

Typeofmetric Examples

Ac#vetest iPerf,IxChariot,EkahauSiteSurvey,AirMagnetSurvey,Speedtestapp,nPerfapp,7signalSapphire,7signalMobileEyeapp

Beaconanalysis/”Wi-Fiscanners”

EkahauSiteSurvey,AirMagnetSurvey,FlukeAircheck,MetageekInSSIDer,Wi-FiExplorer,Wi-FiSignal,AcrylicWi-Fi,CommView,7signalSapphire

Packetanalysis Omnipeek,MetageekEyeP.A.,7signalSapphire

Networkelementcounter

CiscoPrime,ArubaAirwave,Netscout(NeKlow,SNMP)

Fullpacketcapture APincapturemode,Mac,Wireshark,Omnipeek,7signalSapphire

Spectrumanalysis Wi-Spy/Chanalyzer,EkahauSiteSurvey,CiscoCleanAir,OsciumWi-Pry,7signalSapphire

*Examples,notacomprehensivelist


Wi-Fi Experience Depends on more than Radio

4


Wi-Fi connection setup

§ Beacons, probing process

§ Authentication request/response

§ Association request/response

§ EAP variants / captive portals

§ IP address, multi-phase process

§ DNS

§ Applications, TCP, UDP, HTTP,…

§ Roaming (layers 2/3)

5

Discovery

OpenauthenPcaPon

AssociaPon

802.1X/RadiusorCapPveportal

DHCP

DNS

ApplicaPontraffic

Roaming

L2

L2

L2

L2

L3-4

L4-7

L4-7

L4-7


DNS services§  Success rates and delays§  Primary, secondary, tertiary

6

Performance metrics, active test

h\p://www.soasta.com/blog/page-bloat-average-web-page-2-mb/

DHCP/IP services§  Multiple phases

–  Phase 1: DHCP discovery–  Phase 2: DHCP offer–  Phase 3: DHCP request–  Phase 4: DHCP acknowledged

§  Success rate and time

Association and 802.1X authentication§  Association request§  Association response§  EAP authentication

–  Multiple phases–  Depends on the EAP variant

§  Success rate and time§  Status/reason codes


Association and 802.1X authentication

1.  Started WPA supplicant2.  Started scan3.  Started open authentication4.  Completed open authentication5.  Started association6.  Completed association7.  Connected8.  EAP authentication started

–  EAP method proposed by the server–  EAP selected method –  EAP peer certificate validated

9.  EAP authentication completed

7

ScanPme

OpenauthenPcaPonPme

AssociaPonPme

Phase1

Phase2

Phase3

Phase4

Phase5

A\achPme

*PracPcalstepswhichcanbereportedbyastandardWPAsupplicant



Throughput§  “Goodput”/transfer speed§  TCP, HTTP “connection”§  UDP “connectionless”§  One or multiple connections§  TCP: Slow start & congestion

control must be considered§  High latency – high

throughput network requires long test time

§  “Mbit/s” and success %

Delay/Ping§  Bi-directional delay§  ICMP Echo§  32B typical size§  “ms” and success %

8

Web page load§  Load time and success rate§  Average page >2MB


TCP behavior with different devices

9

h\p://repository.lib.ncsu.edu/dr/bitstream/1840.4/8268/1/TR-2012-6.pdf

Slowstart

CongesPoncontrol


Impact of slow start, iPhone 6

10*Testby7signal


Voice quality§  G.711 (64 kbit/s), G.729 (8

kbit/s), ETSI 06.10 (13 kbit/s)§  Max MOS varies/codec§  MOS score algorithms

–  PESQ - Perceptual Evaluation of Speech Quality

–  POLQA - Perceptual Objective Listening Quality Assessment

–  PSQA - Pseudo-Subjective Quality Assessment

§  R-value–  Based on ITU E-model (G.107)

Video quality§  WMV, H.262/MPEG-2, H.263/

MPEG-4, H.264/MPEG-4, H.265/MPEG-H

§  MOS score algorithms–  PEVQ - Perceptual Evaluation of

Video Quality

11


h\p://www.tamos.com/htmlhelp/voip-analysis/mosandr_factor.htmh\p://what-when-how.com/voip/e-model-based-voice-quality-esPmaPon-voip/


Packet loss§  Packets received/packets

sent * 100%§  Ping, bi-directional§  UDP, uni-directional§  TCP, “no packet loss”§  Unit “%”

Jitter§  Maximum deviation from

average value§  Unit “ms”

12


h\p://www.embedded.com/print/4017837


Active testing, ports and protocols§ Test payload for active test

must pass end-to-end§ Firewalls need to allow

test traffic§ Port numbers can vary

from 0 to 65535§ Firewalls, proxies,

application filtering, web accelerators

13

h\p://resources.intenseschool.com/ccna-prep-fundamentals-of-tcpip-transport-applicaPons-and-security/


Radio retransmissions§  802.11: If packet is not

ACK’ed, a complete packet retry follows

§  Radio is persistent, will try up to 7-25 times for the same packet, same or lower MCS

§  Retry % (alt. 1) = packets with retry bit on/all attempted packets

§  Retry % (alt. 2) = What share of original packets require retry (one or many, no matter)

§  Active & passive test

Data rate/MCSs§  What share of packets use

certain rate/MCS§  Data packets use rate control

always. More reliable to calculate rate usage from them

§  Rate control adjust rates based on quality of link and this indicates well signal conditions experienced by AP and client

§  Rate control algorithms are not standardized in 802.11

§  Active & passive test

14

Performance metrics, active/passive


CRC/FCS errors§  Receiver synchronizes to packet

if the 802.11 header (1 or 6 Mbit/s) CRC is correct

§  After synch, whole packet is transferred

§  Frame Check Sequence, FCS at the end

§  Error rate = packets with correct FCS/all attempted packets

§  Some chipsets do not show other than FCS OK packets => cannot calculate error rate %

§  Dilemma: Include or do not include bad FCS packets?

Channel utilization§  QBSS based utilization–  How the AP experiences the

air utilization?–  Information in beacon IE–  802.11e element–  255 states–  Supported increasingly

§  Packet capture based utilization–  Calculate based on packet

capture–  How much airtime captured

packets require vs. whole capturing time

15

Performance metrics, passive test


Air Utilization: Packet capture based model

16

PLCPtrainingfieldand

header,20us

DatapacketDuraPon=size/rate

AIFS

InterFrameArbitraPonPmeAIFS(usedasanaveragearbitraPonPme)

37-70us/frame

MandatoryIniPalBackOffPme

13.5-67.5us/frame

BackoffWorkingassumpPonhereisthatsomebackoffisalwayspresent.

Airu#liza#on=

PLCPheader/preamblePme+FrametransmissionPme+arbitraPonPme+backoffPme

TotalcapturePme


Beacon availability§  Passive and active scan for

beacon capture§  0% or 100%

Channel number§  Static or automated allocations§  Balanced use of all channels§  Channel #§  # of BSSIDs/channel

17


Signal level§  Signal level (or SINR level)§  Static or automated AP power

levels§  Surveys§  dBm (dB)

Noise level§  Reported by the driver§  Hard to measure accurately

with any Wi-Fi chip§  dBm

Copyright © 2015 7signal Solutions, Inc.18


Frame counts§  Busiest senders§  Busiest channels§  Number of frames/min

Frame sizes§  Average frame size§  Impact of different traffic types§  Impact of fragmentation§  Bytes

Data volume§  Total volume§  Volume/frame type§  Volume/client§  MB/Mbit

Clients/STAs§  Number of clients§  QBSS based (beacon)§  Observed traffic based§  Devices manufacturers/types

–  MAC range/OUI


802.11 Status codes§  802.11 Association Status

codes§  Collected from packet capture§  Provide information about

clients association issues §  For example

–  Code 17: Association denied due to overload of the AP

§  Listed–  https://supportforums.cisco.com/document/

141136/80211-association-status-80211-deauth-reason-codes

–  http://community.arubanetworks.com/t5/Technology-Blog/802-11-Reason-Codes-and-Status-Codes/ba-p/257893

802.11 Reason codes§  802.11 De-authentication

Reason codes§  Collected from packet capture§  Provides information on

reasons for clients leaving the AP

§  For example–  Code 5: Disassociated because AP is

unable to handle all currently associated STAs. The access point is busy, performing load balancing, for example.

19

Performance metrics, active/passive test

h\p://www.aboutcher.co.uk/2012/07/linux-wifi-deauthenPcated-reason-codes/


TCP socket failure codes§  TCP specific codes for failed

connections§  For example

–  Connection refused–  Network unreachable–  Socket read error–  Attach to AP failed–  DHCP failure–  IP address set failure–  Route remove failure–  Host unreachable–  Connection timed out

Spectrum analysis§  Spectrum analysis

–  QCA/Atheros Wi-Fi chip integrated–  TI CC2500 chip–  Cisco CleanAir (“Cognio”) chip–  Bandspeed chip

§  Line graphs§  Water wall graphs§  Density graphs§  3D graphs§  Fingerprinting

20

Performance metrics


How to organize the data?

21


Why graphs are REALLY needed?

22

ANSCOMBESQUARTET:Thesamemean,variance,correla#on&regressionlineLesson:AlwaysVisualizeData

h\p://data.heapanalyPcs.com/anscombes-quartet-and-why-summary-staPsPcs-dont-tell-the-whole-story/


Organizing the data

§ Key Performance Indicators (KPI)–  Test results on time axis–  Visualizations, trending lines

§ Time averages–  Often does not make sense to show all samples–  Time averages improve clarity of the data and help with making

the right conclusions–  Averages: 1min, 10min, 1 hour, 1 day, 1 week, specific period

§ Aggregations–  BSSID, SSID, frequency band, channel, service area–  LAN, Wi-Fi–  Test end points

23


Service Level Agreement, SLA

§ Helps to quickly determine is the service working as required

§ Macro KPIs and SLA tables§ KPI types, consider the difference–  Individual test is success or fail, value 0 or 100%

§  Test success rate–  Individual tests have infinite amount of values, “analog nature”

§  Downlink throughput

§ SLA calculation methods–  Average vs. threshold–  Sample count vs. threshold

24


SLA calculation, average basedCompliancy with different threshold values

25

AVG=21Mbit/s

SLA%=21Mbps/30Mbps=70%

30Mbit/s

SLA%=21Mbps/2Mbps=>100%

5Mbit/s2Mbit/s

SLA%=21Mbps/5Mbps=>100%


SLA calculation, sample count basedCompliancy with different threshold values

26

SLA%=10/25samples=40%

SLA%=19/25samples=76% SLA%=23/25samples

=92%

30Mbit/s

5Mbit/s2Mbit/s


Charting

§ Basic chart types–  Line graphs, bar charts, pie charts

§ X-Y charts–  Correlation mapping

§ Averages–  Geometric average –  Arithmetic average

§ Min, max, percentile (5%, 95%) –  Calculated inside the averaging period sample

§ Cumulative Distribution Functions (CDFs)–  Continuous distribution graph

27

Geo_avg=3√(x13�x23�x33)Arith_avg=(x1+x2+x3)/3


Impact of sampling density

§ Manual or automated, your test data rarely covers everything

28

Time

Passivetest

Measurement Measurement

Time

Measurement Measurement

Ac#vetest

Eventinnetwork

Performanceofnetwork


How to use the data and make conclusions?

29


Collecting the test data

30

Sitesurveys

Oneofftests

Trendanalysis

SLAfailure

Emailreports

Emailalarms

SNMPalarms


Drilling in to issue

SSID

Frequencyband

Servicearea

AP/BSSID

Segment

Clientvendor

Client

31


Spotting the problems from data

Missingbeacons/deadAP

FailingtoauthenPcate

FailingtogetIP

Noe2econnecPvity

Lowthroughput

Highdelay

Highpacketloss/ji\er

32

Toomanyclients/AP

HighairuPlizaPon

Highretries/lowMCS

PoorchannelallocaPons

ConPnuouschannelchanges

Interference

ProblemaPcstatus/reason

codes

Ac#vetest Passivetest


Data from a major University in US

•  Network has been already optimized and works well in most aspects

•  Careful analysis reveals still room for further checks•  Challenging, open environment•  HT20 channels used for maximum capacity•  Example data on observed network performance in the main

student SSID•  Data from the 1st floor with 8 APs, where a lot of students spend

time•  2 weeks period around the time school starts

33


Dashboard Macro KPIs

ConnecPvity• Beaconavailability• Radioa\achsuccessrate• Radioa\achPme• AssociaPonsuccessrate• RadioassociaPonPme•  TCPtestsuccessrate

IPservices•  IPaddressPme•  IPaddresssuccessrate

AuthenPcaPon• OpenradioauthenPcaPonsuccessrate• OpenradioauthenPcaPonPme• Radius:TotalEAPauthenPcaPonsuccessrate

Quality•  TCPDLthroughput•  TCPULthroughput•  PingRTT• WebpageloadPme• VoIPMOSDL• VoIPMOSUL

34


Dashboard, 1st floor1 month period

35

Periodofinterest


SLA: Connectivity and AuthenticationOne hour averages

36

Schooloff Schoolstarts 5GHzdoingwell

2.4GHzstruggleswithload

Schooloff Schoolstarts 5GHzdoingwell,butli\lebelow100%



SLA: Connectivity and Quality One hour averages

37

Schooloff Schoolstarts 5GHzdoingwell


Schooloff Schoolstarts 5GHzdoingwell,butli\lebelow100%



SLA: Table view for 5 GHzDaily averages

38

GoodperformanceRetriesabove10%target

level


SLA: Table view for 2.4 GHzDaily averages

39

Radioa\achnotontargetlevel

Throughputnotontargetlevel

VoicequalityULbelowtarget

Retriesabove10%targetlevel


Traffic volume

40


Station count and channel utilizationAreaBand average

41

70clients/APat5GHz

5clients/APat2.4GHz

60%uPlizaPonat2.4GHz

25%uPlizaPonat5GHz


Station count and channel utilization2.4 GHz/AP specific

42

Upto80%uPlizaPon(AP140)

Max12clients/AP(AP140)


Station count and channel utilization5 GHz/AP specific

43

Upto90%uPlizaPon(AP102)

Max100clients/AP

(AP156).Othershaveclearlyless


Frame volumes, Uplink/Downlink

44

5GHzbandcarriesmuchhighertrafficload/AP


End-to-end performance KPIs

45


TCP throughput, Uplink/Downlink (active)

46

Drop

Drop


Voice quality (MOS) and RTT (active)

47

Peakloadincreases

latencyalotat2.4


Jitter and Packet Loss (active)

48


Probe request and response air utilization

49

Requestsandresponsesinbalance


Beacon air utilization and frame errors

50

Lookinggood:BeaconsconsumeverylimitedairPme


Retransmissions, AP/client (passive)

51

APretriesatbothbandsaround50%-60%

Clientretriesatbothbandsaround30%


Spotting specific issues

52


SLA: Connectivity and Authentication2.4 GHz APs

53

AP156FailingauthenPcaPon

AP127FailingauthenPcaPon

DegradedconnecPvity

AP156FailingconnecPvity

AP127FailingconnecPvity


SLA: IP services and Quality 2.4 GHz APs

Degradedquality

SeveralAPslackingIPcompliancy


IP address success rate and time 2.4 GHz APs

…abitlongdelaytogetIP.SLAsetat5s.

IPisalwaysreceived


Reasons for low IP address score 2.4 GHz APs

Delaygerngoffer:DHCPhostnotresponsive

Significantdelay:DHCPhostnotresponsive


Reasons for low connectivity scores 2.4 GHz: AP127 and AP156 connectivity issues

57

APssend/beaconsreceivedpre\ynormally

AP156&127openauthenPcaPonfails

AssociaPonsuccessratedegraded


Reasons for low connectivity scores 2.4 GHz

58

PartoftheAPsneedclearlylongerPmeforassociaPon


Throughputs, daily minimum

59

2.4GHz

5GHzSignificantslowdownunderload,butremainssPllatmulPMbpsspeed

Significantslowdownunderload,butremainssPllat1Mbpsspeed(AP140)

AP140lowest


2.4 GHz channel usage

60

VastmajorityofthenearbyAPsusechannel1



5 GHz channel usage

61


In-balancedchannelusage,unusedcapacity


Reason codes (passive analysis)

62

“4-wayhandshakePmeouts”withmulPpleAPs

“AuthenPcaPonfailed”withmulPpleAPs,especiallywith

AP140


Plan for action§ Check DHCP server serving certain APs, like 140, 102-2–  Impacts both bands–  Are there two separate DHCP servers being used by measured APs?–  Allocate more DHCP capacity to speed up responses

§ Restart AP156 and AP127–  Observe if open authentication issues disappear–  If not, consider replacing the APs next

§ Balance usage of channels on both bands–  Helps to reduce throughput drops during high load–  Helps to perform successful associations and open authentications

§ Improve especially AP140 (2.4 & 5 GHz) RF performance –  AP140 regularly experiences 60-80% air utilization at 2.4 band–  AP140 also experiences lowest throughputs with active tests in both

bands–  “Authentication failed” codes may be due to RF issues–  => Consider new channel and power settings for better channel

quality and better load balancing between APs

63


Demo

64


Cisco,Apple,Xirrus,…


Thank You!

66

www.7signal.com@VPonwireless

Email: [email protected]

Presentation: go.7signal.com/WLPC2016

Documents

Wi-Fi performance measurements · § What share of packets use ... – How much airtime captured ... Performance metrics, active/passive test hp: