Networking Fundamentals, Troubleshooting and Packet Analysis … · 2 packets transmitted, 0 packets received, 100.0% packet loss frizianz-osx:~ frizianz$ So we can't seem to get

Networking Fundamentals, Troubleshooting andPacket Analysis

Fraser McGlinn@frizianz

[email protected]

Background

● Kiwi Expat – Moved here 6 Months ago fromChristchurch.

● I have been working in the Telecommunicationsindustry for just shy of five years, primarily as aNetwork Engineer. This has also included work on ISPrelated services such as Radius, DNS and NMSMonitoring.

● Always been a Linux fan since a young age, built myfirst Linux PC in 2007 running on old parts I scrapedtogether from the junk pile at the local computerrecycler.

Issue:

User defined problem:Internet isn't working from my Home....

What they actually mean:I can't browse to a website.

First before troubleshooting, we understand a bitabout the networking fundamentals and

underlying networking protocols....

OSI Model (theoretical model)

● Packets start at the bottom of the stack(Physical) and get passed up the stack as theyare processed.

● Each Layer is independent of each other andtakes care of a specific task.

● Troubleshooting should always start at layer 1and work up the stack from there due to thedata flow mentioned above.

On that note...

Layer 1 – Physical

● Layer 1 is all about transmitting raw bits on thewire/radio

● Layer 1 is everything physical – physical wiring,network interfaces, optical transceivers or anythingthat is used to make two devices communicate.

● Layer 1's major functions include:

– Modulation

– Bit synchronization in synchronous serial communication

– Circuit Switching

– Multiplexing

– Forward Error Correction


The physical layer is also concerned with:

● Bit Rate

● Point-to-point, multipoint or point-to-multipoint lineconfiguration

● Physical network topology, for example bus, ring,mesh or star

● Simplex, half duplex or full duplex transmission mode

● Autonegotiation


Below are examples of Layer 1 Protocols (There are much more than this)

● Telephone network modems - V.92● EIA RS-232, EIA-422, EIA-423, RS-449, RS-485● Ethernet physical layer 10BASE-T, 10BASE2, 10BASE5, 100BASE-TX,

100BASE-FX, 100BASE-T, 1000BASE-T, 1000BASE-SX and other varieties● Varieties of 802.11 Wi-Fi● DSL● T1 and other T-carrier links, and E1 and other E-carrier links● SONET/SDH● Optical Transport Network (OTN)● GSM● USB physical layer● Avian Carriers (RFC1149) – Make sure to check this one out!

Layer 2 – Data Link

● There are two sublayersto the Data Link Layer

– MAC – This handles theFraming/De-Framing andinteraction with PHY

– LLC – This handles theNetwork Layer protocolMultiplexing/De-Multiplexing

Layer 2 – MAC (Media Access Control)

● On the sending side, the MAC sub-layer is responsible foradding the frame header and the frame trailer. The frameheader consists of layer 2 addresses (known as MAC address)and a few other fields for control purposes, the frame trailerconsists of the CRC/checksum. Then it sends frames to thePHY.

● On the receiving side, the MAC sub-layer receives frames fromthe PHY and is responsible for accepting each frame, byexamining the frame header. It is also responsible for verifyingthe checksum to conclude whether the frame has come corruptduring transmission.

● CRC/checksum computation and verification is quite resourceintensive and is done by a dedicated piece of hardware (suchas your NIC on a PC)

● The MAC sub-layer is also responsible for Collision Resolution.

Layer 2 – LLC (Logical Link Control)

● The LLC sub-layer is responsible for interfacing with theNetwork layer above by doing L3 protocol multiplexing/de-multiplexing.

● On receiving a frame from the physical layer below, the LLCis responsible for looking at the EtherType and handing overthe datagram to the correct upper layer protocol (de-multiplexing) at the network layer above.

● On the sending side, LLC takes packets from different upperlayer protocols like IP, IPX, ARP etc., and hands it over tothe MAC layer after filling the EtherType in the LLC headerportion of the frame (multiplexing).


Below are examples of Layer 1 Protocols (There are much more than this)

● CDP Cisco Discovery Protocol● Ethernet● Frame Relay● ITU-T G.hn Data Link Layer● HDLC High-Level Data Link Control● IEEE 802.11 WiFi● LACP Link Aggregation Control Protocol● LLDP Link Layer Discovery Protocol● LLDP-MED Link Layer Discovery Protocol - Media Endpoint Discovery● PPP Point-to-Point Protocol● STP Spanning Tree Protocol● Token Ring● VLAN - 802.1Q


● As you can see from the above, the source anddestination MAC addresses and the next layerprotocol type (EtherType).

Each upper layer protocol has a different EtherType,so lets see what it looks like when we throw a VLANin the mix.....


● Notice how we now have a different EtherTypeunder the ethernet frame?

● Also take note of how since VLAN's are a layer2 encapsulation type, it also has a next headerethertype in its header too.


MAC Addresses are used on local switchednetworks only.

Layer 3 - Network

The network layer is responsible for packetforwarding including routing.

Some Network Layer protocols are as follows:

● IPv4 Internet Protocol

● IPv6 Internet Protocol

● IPX

Layer 3 - Network

This is what a IPv4 header looks like...

Has a next Protocol field - assigned fromRFC1700

Layer 3 - Network

This is what a IPv6 header looks like.

IPv6 also includes a Next Header ID

Layer 3 - ARP

How do we learn the next hop mac address from a IPAddress? The answer to this is the Address ResolutionProtocol

● ARP operates at Layer 2, but it allows us to work outforwarding MAC addresses for layer 3 addresses.

Layer 3 - ARP

Layer 3 - ARP

Layer 4 - Transport

I shouldn't need to say that much about thisgiven Linux touches these protocols day in andday out.

Layer 4 is all about the protocols that areencapsulated in IP such as the following (butnot limited to):

● TCP

● UDP

MTU – Maximum Transmission Unit

MTU is defined as the maximum PDU (protocol data unit) sizein bytes that can be transmitted over a network segment.

Examples:

● Ethernet – 1500 Bytes

● Ethernet Jumbo Frames – > 1500 Bytes

● ATM AAL5 – 9180 Bytes

● PPP over Ethernet – 1492 Bytes (Can be more when you useethernet jumbo frames

MTU – Maximum Transmission Unit

This is networking's fundamentals..

How would be troubleshoot this issue to come upwith a problem description to summarize?

What would be the first step?

Is there a layer 1 problem? (Physical Layer)

No, my ADSL connection is up

Moving up the OSI Model, what about Layer 2?

DSL basedtechnology usuallyuses PPP overSomething (being theLayer 2 Protocol)

Where something isusually either ATM orEthernet

Do you have IP connectivity to the remote host?

frizianz-osx:~ frizianz$ ping -c 5 frizianz.com

PING frizianz.com (107.170.218.200): 56 data bytes

64 bytes from 107.170.218.200: icmp_seq=0 ttl=51 time=185.638 ms





--- frizianz.com ping statistics ---

5 packets transmitted, 5 packets received, 0.0% packet loss

round-trip min/avg/max/stddev = 185.214/185.406/185.658/0.199 ms

frizianz-osx:~ frizianz$

Usually the best way to test this is to 'ping' the remote host.

This uses a protocol called ICMP (Internet Control Message Protocol)

Two notes:● Sometimes the remote host blocks ICMP so if you can't ping it you

should always verify if this is the case.● One reason why ICMP is a good thing to use, it by default is quite

small on the wire (usually the packet size is 84 Bytes)

20 bytes for the IPv4 header, 8 Bytes for ICMP header, and 58 Bytes ofICMP payload.

Yes I can ping it!

So where to now?

Based on our troubleshooting so far....

● We know its not a Physical problem.● We know its not a layer 2 problem (PPPoE in this case)● We know its not an IP problem (we can ping it)● So the problem must be within Layers 4-7

In the case of TCP, its super easy to check if youhave remote connectivity

You can telnet to the host and port combination(or use another utility such as nc)

frizianz-osx:~ frizianz$ telnet frizianz.com 80 Trying 107.170.218.200...Connected to frizianz.com.Escape character is '^]'.^]telnet> qConnection closed.frizianz-osx:~ frizianz$

Note the small packet sizes and the usual TCPthree way handshake and three way close.

Since we can connect to the service IP/port we knowthat the service on the far side is configured correctly.

Usually when you get to this level in the OSI stack youonly have a few options:

● Layer 8 problem (user problem)

● Layer 7 firewall denying something specific.

● MTU Problem

Usually the MTU across the Internet is 1500 Bytes(maximum on standard ethernet).

How do you verify what the maximum size packetyou can get across the network is?

What if you can't get a packet of 1500 bytes to theremote side? How do you compensate for this?

To test the maximum size that you can get to theremote end, you'd use an ICMP packet which setsthe Do Not Fragment Bit in the IP header.

● Windows: ping -f -l 1472 8.8.8.8● Linux: ping -M do -s 1472 8.8.8.8● Mac OS X: ping -D -s 1472 8.8.8.8

This sets the ICMP payload size to be 1472 bytes(total IP packet size 1500 bytes as it excludes IP andICMP headers)

frizianz-osx:~ frizianz$ ping -D -s 1472 -c 2 frizianz.com

PING frizianz.com (107.170.218.200): 1472 data bytes

556 bytes from bridge5.fw01.mel.au.iama.geek.nz (10.109.0.254): frag needed and DF set (MTU 1492)

Vr HL TOS Len ID Flg off TTL Pro cks Src Dst

4 5 00 dc05 354a 0 0000 40 01 aef6 10.109.0.1 107.170.218.200

Request timeout for icmp_seq 0

556 bytes from bridge5.fw01.mel.au.iama.geek.nz (10.109.0.254): frag needed and DF set (MTU 1492)

Vr HL TOS Len ID Flg off TTL Pro cks Src Dst

4 5 00 dc05 0cdb 0 0000 40 01 d765 10.109.0.1 107.170.218.200

--- frizianz.com ping statistics ---

2 packets transmitted, 0 packets received, 100.0% packet loss

frizianz-osx:~ frizianz$

So we can't seem to get 1500 bytes through tothe remote host.

This seems like the most likely explanation forthis fault.

Since we are dealing with TCP, it has options todeal with low MTU which for whatever reasonare not being handled by Path MTU Discovery(RFC1191)

TCP – Maximum Segment Size

● The TCP MSS is defined to be the relevant IPdatagram size minus 40 bytes to allow for theTCP header and the IPv4 header.

(RFC1191 sec 3.1)

● This is only set in TCP SYN packets.

● This tells to remote host that this is the largestTCP payload that you can receive. Setting thisshould allow the page to load without thepacket being dropped.

TCP – Maximum Segment Size

Linux:

iptables -A FORWARD -o ppp0 -p tcp --tcp-flags SYN,RST SYN -jTCPMSS --set-mss 1400

Cisco:

interface Dialer 0

ip tcp adjust-mss 1400

Mikrotik

/ip firewall mangle

add action=change-mss chain=forward out-interface=pppoe-client1new-mss=1400 protocol=tcp tcp-flags=syn tcp-mss=1401-65535

Success!

When troubleshooting network faults:

Always start at the physical layer and work yourway up.

Come up with a clear problem description todefine your problem.

After working through layer 1 problems, rememberpacket capture tools are your friend.

If in doubt, run a pcap!

Thanks!

[email protected]@frizianz

mailto:[email protected]

Documents

Networking Fundamentals, Troubleshooting and Packet Analysis … · 2 packets transmitted, 0 packets received, 100.0% packet loss frizianz-osx:~ frizianz$ So we can't seem to get