Troubleshooting Wireless LANs with Centralized Controllersfaculty.ccc.edu/mmoizuddin/CISCO LIVE 2008/AGG/BRKAGG-3011.pdf14511_04_2008_c1 EAP-FAST Authentication Overview RADIUS server

© 2006, Cisco Systems, Inc. All rights reserved.Presentation_ID.scr

1

© 2008 Cisco Systems, Inc. All rights reserved. Cisco PublicBRKAGG-301114511_04_2008_c1 2

Troubleshooting Wireless LANs with Centralized Controllers

BRKAGG-3011


2

© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 3BRKAGG-301114511_04_2008_c1

Troubleshooting Wireless LANs

Technology Refresher

Set up Your Network Right

Make Sure Stuff Basically Works

Get Individual Trouble Spots Fixed Up

Get Individual Clients Fixed Up

Nothing to It!


Technology Refresher


3


Wireless LAN technology refresher

802.11/802.1X/WPA

Cisco Unified Architecture/LWAPP

Cisco Unified client mobility

Radio Resource Management


WLAN Topologies – Single 802.11 AP

STA 1

STA 3STA 2

Channel 11


4


WLAN Topologies—InfrastructureMultiple cells

Each cell operates on its own channel

Each AP transmits beacons advertising its BSSID (radio MAC)

All APs offer the given service using the same ESSID (“SSID”)

channel 11channel 6


Steps To Building an 802.11 Connection

1. Listen for Beacons

2. Probe Request

3. Probe Response

4. Authentication Request

5. Authentication Response

6. Association Request

7. Association Response

8. (Optional: EAPOL authentication)

9. (Optional: encrypt data)

10. Move user data

State 1:Unauthenticated,

Unassociated

State 2:Authenticated,Unassociated

State 3:Authenticated,

Associated

802.11 assoc complete


5


802.11 Association Overview (with WPA IE)Station Access Point

Beacon (WPA-IE)

Probe Response (WPA-IE)

Authentication Response

Association Response

Probe Request

Association Request (WPA-IE)

Authentication Request


802.1X authentication – dynamic WEPServer

EAP-ID-Request

Rest of the EAP Conversation

Radius-Access-Accept

(key)EAP-Success

EAPOL-START

EAP-ID-ResponseRADIUS (EAP-ID_Response)

SupplicantAuthenticator

The Supplicant Derives the Session Key from User Password or Certificate and Authentication Exchange

Session Key


6


EAP-FAST Authentication Overview

RADIUS server

EAPOL Start

EAP-Request/Identity

EAP -Response/Identity (EAP-ID)

EAP success

RADIUS Access request

Start EAP AuthenticationAsk client for identity

Access Request with EAP-ID

Perform sequence defined by EAP-FAST

Client derives PMK

keykeyRADIUS Access Accept

(Pass PMK to AP)

Supplicant

EnterpriseNetwork

Secure Tunnel (via TLS & PAC)

Client-side Authentication

AP

WPA Key Management

Protected DATA Transfer

Phase 1

Phase 2


Review: Cisco’s Unified Architecture

Cisco Centralized WLAN Model

Split MAC and Local MAC

LWAPP Architecture (Layer 3 LWAPP)(Layer 2 LWAPP is going away)

Mobility—Layer 2 and Layer 3

Radio Resource ManagementDynamic Channel Assignment (DCA)

AutoRF

Coverage Hole Detection (CHD)


7



Ingress/Egress Point from/to Upstream

Switched/Routed Wired Network (802.1Q Trunk)

Control MessagesData Encapsulation

Access Points Are “Lightweight”—Controlled by a Centralized WLAN Controller

Much of the Traditional WLAN Functionality Moved from Access Points to Centralized WLAN Controller

LWAPP Defines Control Messaging and Data Encapsulation Between Access Points and Centralized WLAN Controller

Lightweight Access Point

Wireless LAN Controller

LWAPP Tunnel

Switched/Routed Wired Network









Remote RF Interface

Real-time 802.11 MAC

RF Spectral Analysis

WLAN IDS Signature AnalysisSecurity ManagementQoS Policies EnforcementCentralized Configuration, Firmware ManagementNorthbound Management Interfaces

LWAPP Carries All Communication Between Access Point and ControllerL2 or L3 TransportMutual Authentication—X.509 Certificate BasedLWAPP Control AES-CCM EncryptedData Encapsulation

Radio Resource Management

Mobility Management

LWAPP Tunnel


8


Division of Labor—Split MAC







Real-time 802.11/MAC Functionality:Beacon GenerationProbe ResponsePower Management/Packet Buffering

Data Encapsulation/De-encapsulationFragmentation/De-fragmentation

802.11e/WMM Scheduling, QueueingMAC Layer Data Encryption/Decryption802.11 Control Messages

Non Real-Time 802.11/MAC Functionality:Association/Disassociation/Reassociation802.11e/WMM Resource Reservation

802.11 Distribution ServicesWired/Wireless Integration Services

802.1X/EAPKey Management

LWAPP Tunnel


Division of Labor—Split MAC Illustrated

802.11 Beacon

Probe Request

Probe Response

802.11 Authentication/Association

802.1X Authenticationand 802.11i Key Exchange

802.11 Data

Probe Is Processed by the AP and Forwarded to

the Controller

802.11 Action Frames

Encryption/Decryptionof RF Packets

Handled at the AP

Add Mobile(AES-CCMP, PTK)

LWAPP Tunnel

Add Mobile(Cleartext, 802.1X Only)


9


Layer 3 LWAPP Architecture

Access points require IP addressing

APs can communicate with WLC across routed boundaries

L3 LWAPP is more flexible than L2 LWAPP and all products support this LWAPP operational ‘flavor’



Data Encapsulation—UDP 12222Control Messages—UDP 12223



LWAPP Tunnel

Layer 2/3 Wired Network—Single or Multiple Broadcast Domains


Layer-2 Roaming—Inter-Controller

Client must be re-authenticated and new security session established

Client database entry moved to new controller

No IP address refresh needed

L2 Inter-Controller roam happens when an AP moves association between APs joined to the different controllers but client traffic bridged onto the same subnet


10


Layer-3 Roaming—Inter-controller

L3 Inter-Controller roam happens when an AP moves association between APs joined to the different controllers but client traffic bridged onto different subnet

Client reauths

Client database entry copied to new controller

Original controller is “anchor”; new one is “foreign”

EoIP tunnel automatically established

No IP address refresh needed

Asymmetric traffic path established – or

Symmetric traffic path

“Anchor”

“Foreign”


Radio Resource Management refresher

Dynamic Channel Assignment (DCA)Selects channels for the radios to use

Responds to interference

AutoRF (Dynamic Power Control, DPC)Reduces radio power, to ensure that each radio hears exactly 3 others at or above the tx-power-thresh value

Coverage Hole Detection (CHD)Detects “coverage holes”, by identifying clients from which we are receiving a poor signal, and accordingly increases radio power, to compensate


11


Set up Your Network Right


Build out your infrastructure

Use the right wired network

RRM tuning tips

bonus! WLC Config Checker

Get your APs to join


12


Wired Network Requirements—AP to WLC (LWAPP Path)

Network RTT - <= 100msec, bandwidth >= 128kbps

Network path must be able to pass IP fragments (but never generate a fragment < 32 bytes)

Network path must not deliver IP fragments via multiple links

APs can be NATted, but WLCs cannot

Trust/set QoS marking as needed for voice

LWAPP

AP WLC


LAG Can’t Reassemble Fragments from Multiple Ports

No IP fragments < 32 bytes (CSCsh96186)

All fragments of any IP datagram must arrive on the same port

src-dst-ip is recommended

src-dst-port will lose

Network must not load balance packets into different LAG ports

4404 subsystem

Link Aggregation

Bundles

4404 subsystem

WiSM

4404

Link AggregationBundle

4402

Link aggregation bundle


13


Wired Network Requirements—WLC to WLC Mobility Tunnel (EoIP Path)

NAT can be used in the EoIP path (as of 4.2.61.0)

Network path must be able to pass 1500B IP packets unfragmented(CSCsm05607)

Workaround: configure wireless devices to use a small MTU

Partial workaround: use ip tcp adjust-mss

Internet

DMZ Corporate Network

Ethernet in IP Tunnel

LWAPPEncapsulation

LWAPPEncapsulation

ip tcp adjust-mss 1300


Radio Resource Management - autoRF

config advanced 802.11b tx-power-control-thresh is the master fader for radio power (values in -60 to -80dBm—lower values for denser installations)

thresh-68

thresh-73


14


Radio Resource Management – detune CHD

Detune Coverage Hole Detection if too many APs are at power 1 in a dense environment (“sticky client” problem)

shrink Coverage threshold (e.g. to 6 dB)

boost Min Clients (e.g. to 5)

See “Radio Resource Management under Unified Wireless Networks”, Document ID 71113, cisco.com

I can’t hear this client too well –

better boost my power!


WLC Config Checker

Windows GUI program, analyzes the output of show run-config

Use config paging disable

Console at 115200 bps, or telnet/ssh

Try to hit return right away when prompted

Provides warnings about the configuration

Displays key aspects of the configuration, and of AP RF info

On CCO, in wireless software downloads area [?]


15


WLC Config Checker—Warnings


WLC Config Checker—AP Nearby Info


16


AP Join troubleshooting

First, the AP must Hunt for the IP addresses of possible WLCs to join

Next, the AP sends Discover messages to all the WLCs, to find out which ones are alive

Then the AP picks the best WLC and tries to Join it

For details, see “Lightweight AP (LAP) Registration to a Wireless LAN Controller (WLC)”, Document ID 70333, cisco.com


LWAPP AP State MachineAP runs HUNTINGalgorithm to find

candidate controllers to join


17


L3 LWAPP WLC Address Hunting

1. LWAPP Discovery broadcast on local subnetcan use ip helper-address, ip forward-protocol

2. Over-the-Air Provisioning (OTAP)

3. Locally stored controller IP addresses

4. DHCP vendor specific option 43 (IP Address should be “Management Interface” IP)

5. DNS resolution of “CISCO-LWAPP-CONTROLLER.localdomain”(should resolve to the “Management Interface” IP)

6. If no controller found, start over

AP Goes Through the Following Steps to Compile a Single List of WLAN Controllers

Note: The Actual Order of This Process Is Irrelevant Because Each AP Goes Through All Steps Before Proceeding to the Next Phase


L3 LWAPP WLC Discovery

Discover

Discover

Discover

Discover

X

Discover

AP tries to send Discover messages to all the WLC addresses that its Hunting process turned up


18


L3 LWAPP WLAN Controller Discovery Algorithm

Once a list of WLAN Controllers is compiled, the AP sends a unicast LWAPP Discovery Request message to each of the controllers in the list

WLAN Controllers receiving the LWAPP discovery messages respond with an LWAPP Discovery Response

LWAPP Discovery Response contains important information:

Controller name, controller type, AP capacity, current AP load, “Master Controller” status, AP-Manager IP address

AP waits for its “Discovery Interval” to expire, then selects a controller and sends an LWAPP Join Request to that controller


WLAN Controller Selection Algorithm

1. If the AP has been configured with primary, secondary, and/or tertiary controller, the AP will attempt to join these first (this is resolved in the Controller “name” field in the LWAPP Discovery Response)

2. Attempt to join a WLAN Controller configured as a “Master” controller

3. Attempt to join the WLAN Controller with the greatest excess AP capacity

The AP Selects the Controller to Join Using the Following Criteria

Note: This Last Step Provides the Whole System with Automatic AP/WLC Load-Balancing Functionality


19


WLAN Controller Join Process—Mutual Authentication – stress SSC

AP LWAPP Join Request contains the AP’s signed X.509 certificateWLAN Controller validates the certificate before sending an LWAPP Join Response

Manufacture Installed Certificate (MIC)—Cisco 1000 Series, all Cisco Aironet APs manufactured after July 18, 2005Self-Signed Certificate (SSC)—LWAPP Upgraded Cisco Aironet APs manufactured prior to July 18, 2005SSC APs must be “authorized” on the WLAN Controller

If AP is validated, the WLAN Controller sends the LWAPP Join Response which contains the controller’s signed X.509 certificateIf the AP validates the WLAN Controller, it will download firmware if necessary and then request its configuration from the WLAN controller

Note: In the Configuration Information, the WLC Includes the IP Addresses of All Other Controllers in Its Mobility List. APs Then Send LWAPP Discovery Messages to Those WLCs, As Well


Troubleshooting LWAPP-Based APs

Can the AP and the WLC communicate?

Make sure the AP is getting an address from DHCP (check the DHCP server leases for the AP’s MAC address)

If the AP’s address is statically set, ensure it is correctly configured

Try pinging the AP from the controller

If pings are successful, ensure the AP has at least one method by which to discovery at least a single WLC

Console or telnet/ssh into the controller to run debugs

Check the Basics First


20


Successful LWAPP AP Join

(WLC_CLI) >debug mac addr 00:0b:85:54:ce:00

(WLC_CLI) >debug lwapp events enable

Received LWAPP DISCOVERY REQUEST from AP 00:0b:85:54:ce:00 to 00:0b:85:40:4a:c0 on port '29'

Successful transmission of LWAPP Discovery-Response to AP 00:0b:85:54:ce:00 on Port 29

Received LWAPP JOIN REQUEST from AP 00:0b:85:54:ce:00 to 06:0a:20:20:00:00 on port '29'

LWAPP Join-Request MTU path from AP 00:0b:85:54:ce:00 is 1500, remote debug mode is 0

Successfully transmission of LWAPP Join-Reply to AP 00:0b:85:54:ce:00

Register LWAPP event for AP 00:0b:85:54:ce:00 slot 0

Received LWAPP CONFIGURE REQUEST from AP 00:0b:85:54:ce:00 to 00:0b:85:40:4a:cb


Failed LWAPP AP Authentication

(WLC_CLI)>debug mac addr 00:12:80:ad:7a:9c(WLC_CLI)>debug lwapp events enable [TIME]: Received LWAPP DISCOVERY REQUEST from AP 00:12:80:ad:7a:9c

to ff:ff:ff:ff:ff:ff on port '1'[TIME]: Successful transmission of LWAPP Discovery-Response to AP

00:12:80:ad:7a:9c on Port 1[TIME]: Received LWAPP JOIN REQUEST from AP 00:12:80:ad:7a:9c to

06:0a:10:10:00:00 on port '1'[TIME]: LWAPP Join-Request does not include valid certificate in

CERTIFICATE_PAYLOAD from AP 00:12:80:ad:7a:9c.[TIME]: Unable to free public key for AP 00:12:80:AD:7A:9C [TIME]: DEBU CTRLR spamProcessJoinRequest:1574

spamProcessJoinRequest : spamDecodeJoinReq failed


21


Set the WLC’s Time

Make sure each controller has the correct time set

Check the WLC’s time:(WLC_CLI) >show time

Manually set the time:(WLC_CLI) >config time manual <MM/DD/YY> <HH:MM:SS>

Or, use NTP: (WLC_CLI) >config time ntp server <Index> <IP Address>

(WLC_CLI) >config time ntp interval <3600 - 604800 sec>

The #1 Reason APs Fail to Join Is Inaccurate Controller Time

Note: NTP Is Not a Quick Fix Because It Is Only Invoked at Controller Boot and at the NTP Interval


Taking Care of SSCs

Each AP has been upgraded properly with the correct time (this date is like your favorite beer’s born-on date)

Each WLC is configured to allow SSC AP authentication

Each WLC has each AP’s SSC hash

Display the input SSC hashes and whether SSC support is enabled:(WLC_CLI) >show auth-list

Enable SSC support:(WLC_CLI) >config auth-list ap-policy ssc enable

Input each AP’s hash:(WLC_CLI) >config auth-list add ssc <MAC Address> <40-bit SSC Hash>

Note: If You’re Not Sure Whether an AP Has Created an SSC, in the AP’s CLI, Do a #show crypto ca certificates. Any Output Indicates that Your AP’s Got Its Certificates

If Controller Time Is Correct, but SSC APs Fail to Join, Make Sure:


22


What If the AP’s SSC Hash Is Missing?(WLC_CLI) >debug mac addr 00:12:80:ad:7a:9c(WLC_CLI) >debug pm pki enable[TIME]: * sshpmGetIssuerHandles:1427 sshpmGetIssuerHandles: locking ca cert table[TIME]: * sshpmGetIssuerHandles:1435 sshpmGetIssuerHandles: calling x509_alloc() for user cert[TIME]: * sshpmGetIssuerHandles:1439 sshpmGetIssuerHandles: calling x509_decode()[TIME]: * sshpmGetIssuerHandles:1458 sshpmGetIssuerHandles: <subject> L=San Jose, ST=California, C=US, O=Cisco Systems[TIME]: * sshpmGetIssuerHandles:1461 sshpmGetIssuerHandles: <issuer> L=San Jose, ST=California, C=US, O=Cisco Systems[TIME]: * sshpmGetIssuerHandles:1471 sshpmGetIssuerHandles: Mac Address in subject is 00:12:80:ad:7a:9c[TIME]: * sshpmGetIssuerHandles:1508 sshpmGetIssuerHandles: Cert is issued by Cisco Systems.[TIME]: * sshpmSsUserCertVerify:1212 ssphmSsUserCertVerify: self-signed user cert verfied.[TIME]: * sshpmGetIssuerHandles:1667 sshpmGetIssuerHandles: getting cisco ID cert handle...<SNIP> Self-Signed Certificate Check </SNIP>[TIME]: * sshpmGetCID:1932 sshpmGetCID: failed to find matching cert.[TIME]: * sshpmGetIssuerHandles:1775 sshpmGetIssuerHandles: SSC Key Hash is 98936bf9c90b30bf3c6bb9a0d7b23668887af49a* <SNIP> DEBU CTRLR </SNIP>

The Fix:(WLC_CLI) >config auth-list add ssc 00:12:80:ad:7a:9c 98936bf9c90b30bf3c6bb9a0d7b23668887af49a

In the WLC GUI, Go to: Security | AAA AP Policies


Does Regulatory Domain Matter? Yes!(WLC_CLI) >debug mac addr 00:12:80:ad:7a:9c(WLC_CLI) >debug lwapp events enable[TIME]: * spamVerifyRegDomain:6202 AP 00:12:80:ad:7a:9c

80211bg Regulatory Domain (-A) does not match with country (JP)reg. domain -JP for slot 0

[TIME]: DEBU CTRLR spamVerifyRegDomain:6167 spamVerifyRegDomain RegDomain set for slot 1 code 0 regstring -A regDfromCb -J

[TIME]: * spamVerifyRegDomain:6202 AP 00:12:80:ad:7a:9c 80211a Regulatory Domain (-A) does not match with country (JP) reg. domain -JP for slot 1

[TIME]: DEBU CTRLR spamVerifyRegDomain:6210 spamVerifyRegDomain AP RegDomain check for the country JP failed

[TIME]: * spamProcessConfigRequest:1730 AP 00:12:80:ad:7a:9c: Regulatory Domain check Completely FAILED. The AP will not be allowed to join.

The fix?Make sure you match your APs’ regulatory domain with your WLCs’.

How do you know how to make sure you do?Search CCO for “Wireless LAN Compliance Status”

Note: In the US, Your APs’ Regulatory Coding Is ‘– A’, not ‘– N’!!!


23


AP Join Problem – Path MTU/Firewall

X

1

43 2

4’

1. AP sends Discover packet (small) – gets thru

2. WLC sends Discover response (small) – gets thru

3. AP sends Join packet (small) – gets thru

4. WLC sends Join response (BIG) – first fragment gets thru, but

4’. The second fragment is blocked by the firewall


When Does AP Fail Over? What Happens Then?

APs will failover to other WLCs if the LWAPP control plane is interrupted

After either:

A missed heartbeat to WLC (sent every 30 seconds)

Or

A Non-ACK’d LWAPP control packet

Then:

The AP will send five successive heartbeats (each a second apart)

If no reply is received, the AP/WLC path is assumed down and the AP will attempt to join another controller

Likely causes:Wired network transmission problem

AP/WLC bug (key rotation problem, buffer leak)


24


Get Your APs to Join

So, make sure ALL WLCs in the cluster will properly allow all APs to join

Make sure all WLCs run the same software version

Make sure all WLCs are set to the correct time

Make sure all WLCs have all upgraded APs’ SSC hashes

See cisco.com Document ID: 99948, “Troubleshoot a Lightweight Access Point Not Joining a Wireless LAN Controller”


Q and A


25


Make Sure Stuff Basically Works


Make Sure It Basically Works

Turn it on and see if anyone complains

Walk around with a PC running a continuous ping to the default gateway

Walk around with a 7921 in a call to a 7960?Push the ? button the 7920 twice—monitor the RTP stats in real time

Set your PC’s interface MTU to the max, and download giant files


26


Get Trouble Spots Fixed Up


Fix up Trouble Spots

Reproduce problem at the application layer

Look at the area with a site survey type tool

Check for interference (SpEx)

Eliminate interference sources

Hand-tune RRM

Change or reorient antennas

Move or add APs


27


My Poorman’s Method

laptop with CB21AG, free Cisco Aironet Site Survey Utility, windump installed

have a continuous ping –t running to my default gateway

write all packets to/from the wireless adapter to a file:c:\>windump –p –i 3 –w d:\tmp\windump.enc

(interface # comes from windump –D)

In CASSU, start AP scan logging (log updates every five seconds)

now, walk around and make a note of where I am at times of high latency/packet loss


Poorman’s Method—CASSU


28


Poorman’s Method—Ping

Now, look in the packet capture (filtering on ICMP), and find the corresponding high latency times

74 ms latency!

74 ms latency!


Poorman’s Method—CASSU Scan Log

Now, look in the CASSU scan log (SST_APScanLog.txt) to see what the APs looked like, at the time of concern (17:50:28)

I can ignore all the BSSIDs except the ones advertising my SSID of interest

Hm ... no APs above -85dBm—I bet VoIP won’t work too well here ...

2008-04-27 17:50:29 ,blizzard,00:19:A9:41:AC:B0,-86,Secure,G,11 (2462),54,tuc-00-ap3,0,4,"CAC, CEC, QBSS, WMM, WPA, RM-Normal"2008-04-27 17:50:29 ,blizzard,00:19:A9:41:13:90,-85,Secure,G,11 (2462),54,tuc-00-ap1,0,4,"CAC, CEC, QBSS, WMM, WPA, RM-Normal"2008-04-27 17:50:29 ,blizzard,00:19:A9:41:0B:F0,-85,Secure,G,1 (2412),54,tuc-00-ap2,0,4,"CAC, CEC, QBSS, WMM, WPA, RM-Normal"


29


Q and A


Fix Individual Clients


30


Hand to Hand Combat with Clients

Client can’t talk to the networkAnatomy of a ping

“Autopsy tools”

“Random” client disconnects


DHCP

What it takes for your wireless client to ping (assuming EAP and DHCP, with L3 mobility configured)

Client probes for the SSIDClient authenticates/associates in 802.11 to an APEAP does its thingDHCP succeedsClient reaches RUN stateIP packet successfully transmitted by client over the air to AP, tunneled in LWAPP to WLC, (forwarded in EoIP to anchor WLC), decapsulatedby WLC to the wired network, forwarded to IP peer, and back again

IP

WLC

WLC

IP

IP

ACSLWAPP EO

IP

802.11 data

802.11 data

802.11 management

LWAPP802.11 management

RADIUSsupp.

driver

radio

EAPchan. 1

chan. 11

Anatomy of a Ping


31


What it takes for me to figure out why your wireless client can’t ping!

IP

WLC

WLC

IP

IP

ACSLWAPP EO

IP

802.11 data

802.11 data

802.11 management


RADIUS

supp.driver

radio

EAP

Supplicantlogs

Driverdebugs/ adapter capture

chan. 1

chan. 11

wirelesssniff

wirelesssniff

APdebugs

wiredsniff

WLCdebugs

wiredsniff

ACSlogs

DHCP

DHCPlogs

NTP

Spectrum analysis

Autopsy of a Ping


<digression> Autopsy tools


32


Supplicant logs are needed to figure out what the client EAP supplicant is thinking. How to turn them on, and what they say, is entirely supplicant-specific.

WZC supplicant log:netsh ras set tracing * enabled —logs in c:\windows\tracingsee http://www.microsoft.com/technet/network/wifi/wlansupp.mspx

PROSet supplicant log: under hklm\software\intel\wireless\settings1xconfigdbg=wwxyz; 1xDebugLevel=dword:0x18;1xLogLevel=dword:0x18logs in c:\ (subject to change without warning)

ADU: see CSCsi16921CSSC: see Log Packager utility on cisco.com

IP

WLC

WLC

IP

IP

ACSLWAPP EO

IP

802.11 data

802.11 data

802.11 management


RADIUSsupp.driver

radio

EAP

Supplicantlogs

chan. 1

chan. 11DHCP

Autopsy Tools—Supplicant


Driver debugs—use only under medical supervision (probably need a special driver build)Client adapter capture—capture a packet trace from the wireless adapter using Wireshark/windump in non-promiscuous mode (Windows)

Shows “Ethernet-II” packets at the NDIS layer (e.g. all IP packets sent to/from this client)Shows EAPOL packets (usually), so very helpful for EAP/supplicant troubleshootingDoes not show 802.11 management frames (no beacons, probes, authentication/association)

IP

WLC

WLC

IP

IP

ACSLWAPP EO

IP

802.11 data

802.11 data

802.11 management


RADIUSsupp.driver

radio

EAPchan. 1

chan. 11DHCP

Driver debugs/adapter capture

Autopsy Tools—Client Driver


33


Wireless packet capture is essential for 802.11 support

Good options (Windows PCs):Omnipeek from Wildpackets (3945 with 10.5.1.75 driver or CB21AG)

Wireshark with CACE Technologies AirPcap adapters

USB adapters nice for multichannel sniff

AirMagnet

CommView for WiFi from Tamosoft

IP

WLC

WLC

IP

IP

ACSLWAPP EO

IP

802.11 data

802.11 data

802.11 management


RADIUSsupp.driver

radio

EAPchan. 1

chan. 11DHCP

Wireless SniffWireless Sniff

Autopsy Tools—Wireless Sniff


Wireless Sniff—Some Tips

One separate packet capture per wireless channel of interest—a capture that scans across multiple channels is only useful for device discoveryAlways perform unfiltered captures unless you know there are no RF issues (need to see beacons, acks)Configure analyzer to cut a new file every 20–30MBConfigure analyzer not to display updated packet list during capture (reduce CPU load and minimize drops)When troubleshooting a roaming client, will need multiple analyzers moving in concert with the client under test (put everything on a cart and get some good exercise)To capture multiple channels at once, the CACE USB 11a/g adapter is a good option—put a bunch of them into the same USB hub and get it all on the same PCNTP sync everything


34


Use spectrum analysis to capture RF spectrum behavior—necessary to identify/track down non-802.11 interference sources

Cisco’s product: Spectrum Expert (nee Cognio)

IP

WLC

WLC

IP

IP

ACSLWAPP EO

IP

802.11 data

802.11 data

802.11 management


RADIUSsupp.driver

radio

EAPchan. 1

chan. 11DHCP

Spectrum Analysis

Autopsy Tools—Spectrum Analysis


SpEx Spectrogram

Very Low Power/Activity

Microwave Oven

Active AP

Mouse Hover Reveals Channel Position,

Sweep Time/Date, and Top Five Devices Color Indicates Power,

Red = -35 dBm

High Power, Bursting over Time


35


SpEx Tips

When capturing, be sure to have an 802.11 adapter installed, enabled, but configured not to associate to a WLAN

Spectrum Expert cannot identify 802.11 devices (MAC address, etc.) without an 802.11 adapter’s aid

NTP sync your SpEx host!


To Collect Debugs from LWAPP IOS APs, You Can:

connect via console (use hidden debug lwapp console cli then conf t to set the console speed to 115200)

from WLC CLI, use:debug ap enable APnamedebug ap command “debug command” APname

in 5.0, can use telnet/ssh to connect to an LWAPP IOS AP

IP

WLC

WLC

IP

IP

ACSLWAPP EO

IP

802.11 data

802.11 data

802.11 management


RADIUSsupp.driver

radio

EAPchan. 1

chan. 11DHCP

LWAPP (IOS) AP Debugs

Autopsy Tools—AP Debugs


36


AP Debugs—Tips

By default, radio debugs (debug dot11 dot11radiox) appear only on the console. To see radio debugs in your telnet/ssh/WLC CLI session, use the command no debug dot11 dot11radiox printf, where x is 0 or 1

Useful radio debugs:debug dot11 dot11radiox trace print mgmt keys beacon rcv xmt (beacon, rcv, xmt apt to be extremely verbose!)

Useful LWAPP join debugs:debug dhcpdebug ip udpdebug lwapp client {config, error, event, packet}


Autopsy Tools—Wired Sniff

When capturing from trunk ports, always capture with 802.1q tags(watch out for packets in the wrong VLANs)

Cut new file every 20/30MB; don’t display packet updates in real time

NTP sync your sniffers!

IP

WLC

WLC

IP

IP

ACSLWAPP EO

IP

802.11 data

802.11 data

802.11 management


RADIUSsupp.driver

radio

EAP

Wired Capture

chan. 1

chan. 11DHCP


37


Autopsy Tools—WLC Debugs

Capture WLC debugs from a telnet/ssh or console (115200 bps) session

Simplest debug for one client under test:debug client MACaddress

enables basic dot11, dot1x, pem and dhcp debugs

IP

WLC

WLC

IP

IP

ACSLWAPP EO

IP

802.11 data

802.11 data

802.11 management


RADIUSsupp.driver

radio

EAP

WLC debugs

chan. 1

chan. 11DHCP


WLC Debugs

More general client debugging options:debug dot11debug dot1xdebug aaa <= use for RADIUS troubleshootingdebug pemdebug mobility <= handoffsdebug dhcpUse debug mac MACaddr to filter on a single client

NTP sync your WLC!


38


Autopsy Tools—RADIUS Logs

See “Logs and Reports” section of the ACS User GuideSystem Configuration -> Service Control -> Full for a deep dive, but beware memory/disk exhaustion—use under medical supervisionNTP sync your ACS!

IP

WLC

WLC

IP

IP

ACSLWAPP EO

IP

802.11 data

802.11 data

802.11 management


RADIUSsupp.driver

radio

EAP

RADIUS Logs

chan. 1

chan. 11DHCP


Autopsy Tools—DHCP Logs

IOS DHCP server:debug ip dhcp server eventsdebug ip dhcp server packet

IP

WLC

WLC

IP

IP

ACSLWAPP EO

IP

802.11 data

802.11 data

802.11 management


RADIUSsupp.driver

radio

EAP

DHCP logs

chan. 1

chan. 11DHCP


39


Autopsy Tools </digression>


Client probes for the SSID

Client authenticates/associates in 802.11 to an AP

EAP does its thing

DHCP succeeds

Client reaches RUN state

IP

WLC

WLC

IP

IP

ACSLWAPP EO

IP

802.11 data

802.11 data

802.11 management


RADIUSsupp.

driver

radio

EAPchan. 1

chan. 11

Probe Req

Probe ReqProbe Req

Probe Req

Probe Resp

DHCP

Anatomy of a Ping—Probing

Probe Resp

wireless sniffwireless sniff


40


Autopsy—Probing

Clients broadcasts a probe for the SSID of interest

AP hopefully unicasts back a probe response

Probe response includes interesting facts (Information Elements) about the service


Problems at Probing Stage

What if the client never sends out a probe?Is it configured for the SSID of interest?

What if the AP doesn’t send back the probe response?Is it (WLC) configured for the SSID of interest?

Do you have RF coverage from this AP? (can you see beacons from it?)

What if the client never moves beyond probing?Does it like the IEs that the AP is sending out? Try different crypto settings; disable Aironet extensions; try different basic rates; etc.


41


DHCP



EAP does its thing

DHCP succeeds


IP

WLC

WLC

IP

IP

ACSLWAPP EO

IP

802.11 data

802.11 data

802.11 management


RADIUSsupp.

driver

radio

EAPchan. 1

chan. 11

Anatomy—802.11 Auth/Assoc

Wireless Sniff WLC Debugs

Association

AuthenticationAuthentication

Association


Autopsy—802.11 Auth/Assoc

Client and AP authenticate to each other (normally just Open authentication nowadays)

Client tries to associate to the AP, hopefully gets a status=0 (successful) response

What if unsuccessful?Check status code

Run debugs on WLC


42




EAP does its thing

DHCP succeeds


IPIP

WLC

WLC

IP ACSLWAPP EO

IP

802.11 data

802.11 data

802.11 management


RADIUSsupp.

driver

radio

EAPchan. 1

chan. 11

EAPOL Start

EAP SUCCESS

EAP ID Response

EAP Request IDRADIUS Access-Request

EAP blah blah blah blah

DHCP

Anatomy—EAP Does Its Thing

Driver Debugs/Adapter CaptureSupplicant

LogsWLC

DebugsRADIUS

Logs

pass keys all around the place


802.11 Capture of 802.1X MS-PEAP (dWEP)


43


Wireshark Capture of MS-PEAP (WPA2)


Successful 802.1X Client Authentication

(WLC_CLI) >debug mac addr 00:13:ce:57:2b:84(WLC_CLI) >debug dot1x events enable [TIME]: * dot1x_auth_txReqId:2827 Sending EAP-Request/Identity to mobile 00:13:ce:57:2b:84 (EAP Id 1)[TIME]: * dot1x_authsm_capture_supp:675 Received EAPOL START from mobile 00:13:ce:57:2b:84[TIME]: * dot1x_handle_eapsupp:1962 Received Identity Response (count=n) from mobile 00:13:ce:57:2b:84<SNIP> Series of 802.1X EAP Requests/Responses </SNIP>[TIME]: * dot1x_process_aaa:898 Processing Access-Challenge for mobile 00:13:ce:57:2b:84[TIME]: * dot1x_bauthsm_txReq:465 Sending EAP Request from AAA to mobile 00:13:ce:57:2b:84 (EAP Id 19)[TIME]: * dot1x_handle_eapsupp:1997 Received EAP Response from mobile 00:13:ce:57:2b:84 (EAP Id 19, EAP

Type 25)[TIME]: * dot1x_process_aaa:906 Processing Access-Accept for mobile 00:13:ce:57:2b:84[TIME]: * createNewPmkCacheEntry:691 Creating a new PMK Cache Entry for station 00:13:ce:57:2b:84 (RSN 0)[TIME]: * dot1x_auth_txCannedSuccess:2594 Sending EAP-Success to mobile 00:13:ce:57:2b:84 (EAP Id 19)[TIME]: * sendDefaultRc4Key:450 Sending default RC4 key to mobile 00:13:ce:57:2b:84[TIME]: * sendKeyMappingRc4Key:325 Sending Key-Mapping RC4 key to mobile 00:13:ce:57:2b:84[TIME]: * dot1x_trans_authsm:2448 Received Auth Success while in Authenticating state for mobile

00:13:ce:57:2b:84

* <SNIP> DEBU STA 00:13:ce:57:2b:84 </SNIP>

debug dot1x events


44


Failed 802.1X Client Authentication

(WLC_CLI) >debug mac addr 00:13:ce:57:2b:84(WLC_CLI) >debug dot1x events enable [TIME]: * dot1x_auth_txReqId:2827 Sending EAP-Request/Identity to mobile 00:13:ce:57:2b:84

(EAP Id 1)[TIME]: * dot1x_authsm_capture_supp:675 Received EAPOL START from mobile

00:13:ce:57:2b:84[TIME]: * dot1x_handle_eapsupp:1962 Received Identity Response (count=n) from mobile

00:13:ce:57:2b:84<SNIP> Series of 802.1X EAP Requests/Responses </SNIP>[TIME]: * dot1x_process_aaa:898 Processing Access-Challenge for mobile 00:13:ce:57:2b:84[TIME]: * dot1x_bauthsm_txReq:465 Sending EAP Request from AAA to mobile 00:13:ce:57:2b:84

(EAP Id 14)[TIME]: * dot1x_handle_eapsupp:1997 Received EAP Response from mobile 00:13:ce:57:2b:84

(EAP Id 14, EAP Type 25)[TIME]: * dot1x_process_aaa:928 Processing Access-Reject for mobile 00:13:ce:57:2b:84[TIME]: * dot1x_auth_txCannedFail:2865 Sending EAP-Failure to mobile 00:13:ce:57:2b:84 (EAP

Id 14)


debug dot1x events—Username/Password Failure


Check Client Record for Details

In the WLC GUI, Go to: Wireless | Clients and Select Details for the Client of Choice


45


Successful 802.1X Client Authentication

(WLC_CLI) >debug mac addr 00:13:ce:57:2b:84(WLC_CLI) >debug aaa events enable [TIME]: * sendRadiusMessage:2494 Successful transmission of Authentication Packet (id 49) to

20.20.20.12:1812, proxy state 00:13:ce:57:2b:84-ce:57[TIME]: DEBU CTRLR processIncomingMessages:3480 ****Enter processIncomingMessages: response

code=11[TIME]: DEBU CTRLR processRadiusResponse:3053 ****Enter processRadiusResponse: response

code=11[TIME]: * processRadiusResponse:3325 Access-Challenge received from RADIUS server 20.20.20.12 for

mobile 00:13:ce:57:2b:84 receiveId = 2[TIME]: * sendRadiusMessage:2494 Successful transmission of Authentication Packet (id 59) to

20.20.20.12:1812, proxy state 00:13:ce:57:2b:84-ce:57[TIME]: DEBU CTRLR processIncomingMessages:3480 ****Enter processIncomingMessages: response

code=2[TIME]: DEBU CTRLR processRadiusResponse:3053 ****Enter processRadiusResponse: response code=2[TIME]: * processRadiusResponse:3325 Access-Accept received from RADIUS server 20.20.20.12 for

mobile 00:13:ce:57:2b:84 receiveId = 2


debug aaa events


Failed 802.1X Client Authentication

AAA connectivity failure will generate an SNMP trap

debug aaa events—AAA Server Unreachable

(Cisco Controller) >debug mac addr 00:13:ce:57:2b:84(Cisco Controller) >debug aaa events enable [TIME]: * sendRadiusMessage:2494 Successful transmission of Authentication Packet (id 66) to 20.20.20.12:1812, proxy state 00:13:ce:57:2b:84-ce:57[TIME]: * sendRadiusMessage:2494 Successful transmission of Authentication Packet (id 66) to 20.20.20.12:1812, proxy state 00:13:ce:57:2b:84-ce:57[TIME]: * sendRadiusMessage:2494 Successful transmission of Authentication Packet (id 66) to 20.20.20.12:1812, proxy state 00:13:ce:57:2b:84-ce:57[TIME]: * sendRadiusMessage:2494 Successful transmission of Authentication Packet (id 66) to 20.20.20.12:1812, proxy state 00:13:ce:57:2b:84-ce:57[TIME]: * sendRadiusMessage:2494 Successful transmission of Authentication Packet (id 66) to 20.20.20.12:1812, proxy state 00:13:ce:57:2b:84-ce:57[TIME]: * sendRadiusMessage:2494 Successful transmission of Authentication Packet (id 66) to 20.20.20.12:1812, proxy state 00:13:ce:57:2b:84-ce:57[TIME]: * radiusProcessQueue:2735 Max retransmission of Access-Request (id 66) to 20.20.20.12 reached for mobile 00:13:ce:57:2b:84[TIME]: * sendAAAError:323 Returning AAA Error 'Timeout' (-5) for mobile 00:13:ce:57:2b:84


In the WLC GUI, Go to: Management | SNMP Trap Logs


46


Verify Complete 802.11/802.1X Connectivity

(WLC_CLI) >debug mac addr 00:13:ce:57:2b:84(WLC_CLI) >debug pem state enable[TIME]: pem_api.c:1780 - State Update 00:13:ce:57:2b:84 from RUN (20) to START (0)[TIME]: pem_api.c:1836 - State Update 00:13:ce:57:2b:84 from START (0) to AUTHCHECK (2)[TIME]: pem_api.c:1859 - State Update 00:13:ce:57:2b:84 from AUTHCHECK (2) to 8021X_REQD (3)[TIME]: pem_api.c:3977 - State Update 00:13:ce:57:2b:84 from 8021X_REQD (3) to L2AUTHCOMPLETE (4)[TIME]: pem_api.c:4152 - State Update 00:13:ce:57:2b:84 from L2AUTHCOMPLETE (4) to RUN (20)

debug pem state


Troubleshooting 802.1X

Make sure the RADIUS server is properly configured

Make Sure the Correct Shared Secret Is Input

Select the Correct RADIUS Port (Common Ports Are 1812 and 1645)

Status Must Be Enabled

Network User Auth Has to Be Enabled for This AAA Server to Be Used

In the WLC GUI, Go to: Security | AAA RADIUS Authentication and Then Select Edit or New


47



Make sure the proper security policy is enabled for both encryption and authentication

Step (1): Select the Desired Layer 2 Security Configuration

Step (2): Ensure at Least One RADIUS Server Is Configured per WLAN. Configure Additional Ones for Fedundancy

In the WLC GUI, Go to: WLANs | WLANs WLANs and Then Select Edit for the WLAN of Interest



Enabled Logging in your ACS server to identify where issues might lie with backend authentication

Make Sure at Least Logging for Failed Attempts Is Enabled on ACS So Server-side Debugging Can Be Performed

In ACS, Select System Configuration | Logging and Enable Each Desired Option


48


IP

Anatomy—DHCP Succeeds

Client probes for the SSIDClient authenticates/associates in 802.11 to an APEAP does its thingDHCP succeedsClient reaches RUN state

IP

WLC

WLC

IP ACSLWAPP EO

IP

802.11 data

802.11 data

802.11 management


RADIUSsupp.

driver

radio

EAPchan. 1

chan. 11DHCP

DHCP discover

DHCP offerDHCP discover

DHCP offer

DHCP requestDHCP request

DHCP ack

DHCP ack


Client IP Provisioning via DHCP

(WLC_CLI) >debug mac addr 00:13:ce:57:2b:84(WLC_CLI) >debug dhcp message enable [TIME]: dhcp option: received DHCP DISCOVER msg<SNIP> DHCP Discover message details </SNIP>[TIME]: Forwarding DHCP packet (332 octets) from 00:13:ce:57:2b:84

-- packet received on direct-connect port requires forwarding to external DHCP server. Next-hop is 20.20.20.1

[TIME]: dhcp option: received DHCP OFFER msg[TIME]: dhcp option: server id = 20.20.20.1[TIME]: dhcp option: netmask = 255.255.255.0[TIME]: dhcp option: gateway = 20.20.20.1<SNIP> DHCP Offer message details </SNIP>[TIME]: dhcp option: received DHCP REQUEST msg[TIME]: dhcp option: requested ip = 20.20.20.113[TIME]: dhcp option: server id = 1.1.1.1<SNIP> DHCP Request message details </SNIP>[TIME]: Forwarding DHCP packet (340 octets) from 00:13:ce:57:2b:84

-- packet received on direct-connect port requires forwarding to external DHCP server. Next-hop is 20.20.20.1

[TIME]: dhcp option: received DHCP ACK msg<SNIP> DHCP Ack message details </SNIP>

debug dhcp message


49


Troubleshooting DHCP

Clients are not configured for static addressing

DHCP scopes are properly configured (either external or internal DHCP)

External servers: Need to support DHCP proxy—if they don’t, turn on bridging:(WLC_CLI) >config dhcp proxy disable

Internal DHCP server: After properly configuring the WLC’s scopes, each interface needs to have the WLC’s management IP as its DHCP server IP address, as below:

In the WLC GUI, Go to: Controller | Interfaces and

Select Edit for the Interface of Choice

For Internal DHCP, Input the WLC’s Management IP Address Here

Note: The WLC’s Internal DHCP Server Will Provide Addresses to APs, As Well, Provided the WLC Is Running 4.0 or Later and the AP DHCP Requests Can Find the Controller’s Management Interface

If Clients Aren’t Getting Addresses Properly via DHCP, Ensure:


Anatomy—PING Succeeds!!



EAP does its thing

DHCP succeeds


IP

WLC

WLC

IP

IP

ACSLWAPP EO

IP

802.11 data

802.11 data

802.11 management


RADIUSsupp.driver

radio

EAPchan. 1

chan. 11DHCP


50


ACS

DHCP

Now for the Hard Part—Troubleshooting Roaming

Take this ...

And move the client under test through it, inthree dimensions, in real time!

IP

WLC

WLC

IP

IP

LWAPP EOIP

802.11 data

802.11 data

802.11 management


RADIUSsupp.driverradio

EAPchan. 1

chan. 11


Troubleshooting a Roaming Client in SituIs Very Hard—You Don’t Want to Do This


51


So Instead, Factor Out Variables

Does the roaming problem happen when using open auth/no crypto? (Factor out: EAP, CCKM, PMK caching, etc.)

Does the roaming problem with intracontroller roams, or only intercontroller? (Factor out: VLAN configproblems, CAM table, L3 mobility problems, mobility group config problems)

Does the roaming problem occur only in specific locations? (Factor out: RF coverage issues)

Does the problem happen when using one supplicant, but not another? (Factor out: specific supplicant issues)


Do Some Clients Have Roaming Problems, and Others Not?

Try tuning the client roaming behaviorIntel: Roaming Aggressiveness knob

7921: lock to 802.11a if you have the coverage

CB21ag: turn down Scan Valid, BSS Aging in Device Manager (see “Optimize CB21AG/PI21AG Roaming Behavior”, Document ID 69403, cisco.com)

Try upgrading the client code7921: must have at least 1.0.5

Intel: drivers in latest (April ’08) 11.5.1.2 bundle


52


What If Some Clients Just Don’t Roam Right, No Matter What?

Prove that another wireless adapter (CB21AG?) in the identical application, works fine

Escalate to your laptop/device vendor

Open a case with Cisco, if TAC assistance is needed in setting up the back end debugging


Q and A


53


Recommended Reading

802.11 Wireless Networks, 2nd Ed., Matthew Gast, O’ReillyReal 802.11 Security, Edney and Arbaugh, Addison-Wesley“Radio Resource Management under Unified Wireless Networks”, Document ID 71113, cisco.com“Troubleshoot a Lightweight Access Point Not Joining a Wireless LAN Controller”, Document ID: 99948, cisco.com


Complete Your Online Session Evaluation

Give us your feedback and you could win fabulous prizes. Winners announced daily.

Receive 20 Passport points for each session evaluation you complete.

Complete your session evaluation online now (open a browser through our wireless network to access our portal) or visit one of the Internet stations throughout the Convention Center.

Don’t forget to activate your Cisco Live virtual account for access to all session material on-demand and return for our live virtual event in October 2008.

Go to the Collaboration Zone in World of Solutions or visit www.cisco-live.com.


54


Documents

Troubleshooting Wireless LANs with Centralized Controllersfaculty.ccc.edu/mmoizuddin/CISCO LIVE 2008/AGG/BRKAGG-3011.pdf14511_04_2008_c1 EAP-FAST Authentication Overview RADIUS server