2015 03 06 lmtv wtf http webcast

© 2015, The Technology Firm www.thetechfirm.com

Baseling/Troubleshooting HTTP

Getting Started

From Throughput & Latency Course

Tony Fortunato, Sr Network Performance Specialist

www.thetechfirm.com

© 2015, The Technology Firm WWW.THETECHFIRM.COM

Notes

Nothing beats attending a live event, but if you can’t make it, head to www.lovemytool.com for the replay.

Check with www.lovemytool.com for future dates and topics.

Use the Q&A icon to post questions and I will answer them as I see them. I will also leave some time at the end as well.

Don’t forget to Like the video on youtube when you find a topic interesting. I use this to determine future topics and always appreciate your feedback.


About Your Presenter

Tony Fortunato is a Sr Network Performance Specialist with The Technology Firm (www.thetechfirm.com) who has experience with training, designing, implementing, and troubleshooting networks since 1989.

Tony will teach or troubleshoot on your network, with your staff and your tools as part of his customized onsite training service.


A Little bit about HTTP

HTTP is • Used to send or receive data

HTTP is not• Just for surfing the web• Reserved for web browsers• Just used over TCP port 80

For the purposes of this session, we will focus on the HTTP/WEB browsing operation.


Sample Test Configuration

5

Go to www.thetechfirm.com

No Capture filter

www.thetechfirm.com

10.44.10.171

Internet

Tip: To quickly eliminate all local traffic, simply filter on all traffic to and from your

router’s MAC address.


Anatomy of a HTTP Connection

6

Wireshark

DNS

ARP

TCP

HTTP


DNS

You can start with a basic DNS display filter to get an idea of how many DNS packets are required to build your webpage.

For specific DNS analysis, filter on a specific DNS transaction ID.

7

In this example, it took 28ms to resolve this domain name.

Wireshark

Wireshark


DNS

If you would like to find out how many DNS requests were sent, then you can use this Wireshark display filter

This is a great way of documenting how many external references a webpage has• i.e More DNS references, more external dependencies, more time to build

Also helpful to see which browser extensions are loaded• For example; bing search engine will have bing DNS name looksup

Wireshark


DNS

9

If you need to analyze or check DNS response time frequently, you may find manually performing this exercise very time consuming.

That’s when you can look for products that will do this for you or figure out a better way of efficiently figuring it out

Network Instruments Observer


DNS - Errors

10

Since DNS is UDP based, no error messages are typically returned if the server is completely down

In some cases, if just the DNS service is down, you might see an ICMP port unreachable message from the server

Packet 1 you can see the lookup to 1.0.0.0 goes unanswered Packet 6 My computer tries to lookup dns.msfncsi to the same DNS ip address, what

is msfncsi? Hang on and you’ll see.. Packet 7 we can see my computer use its secondary DSN of 8.8.8.8 Packet 8 we can see the DNS reply from 8.8.8.8

In this example it took over 6.5 seconds before trying the secondary DNS

Wireshark

Windows Event Viewer


DNS – ICMP Messages By filtering on ICMP and DNS you can identify if there are any underlying issues

11


Not all DNS Servers Are Created Equal Not all servers are created equal which also applies to the path to and from these

servers/networks

In this example you can see that 8.8.8.8 has better performance compared to 10.44.10.94, the local server

12

Local DNS

Remote DNS

The client was very close to improving performance by using a local DNS server, but this local server did not cache any of the name lookups, so it wasn’t doing much to improve performance.

Don’t forget about LDAP, NetBIOS/WINS or other name servers you may be using


Improving DNS Performance - Example In my office, I enable the DNS service on my Cisco router so it can cache common domain

names we access (i.e. email, google, youtube, thetechfirm, etc..) It was pretty easy to enable

• Router1# configure terminalRouter1(config)# ip dns serverRouter1(config)# ip domain-lookupRouter1(config)# ip name-server 8.8.8.8

You can even add your own local hosts• Router1(config)# ip host churchill 10.44.10.94

I took a trace before and after to document the difference.

13

Router with no ip dns enabled

Router with ip dns enabled first query

Router with ip dns enabled second query


Active Probing - MSFTNCSI

Microsoft is probing a certain address (msftncsi.com and dns.msftncsi.com) to check your internet connection and display an “active” network icon. It’s also probing

NOTE: If you prevent NCSI (Network Connection Status Indicator) from connecting to http://www.msftncsi.com, applications that perform checks for the existence of Internet connectivity might work more slowly. Also, if a computer is brought into a hot spot that requires sign-in, the computer might not detect that hot spot.

You can disable Active Probing either to stop these DNS queries.

14


Stop NCSI Packets by Changing a Registry Setting

For best results, close all programs on the computer on which you are changing the registry setting.

To open a command prompt as an administrator, click Start, point to All Programs, click Accessories, right-click Command Prompt, and then click Run as administrator.

If the User Account Control dialog box appears, confirm that the action it displays is what you want, and then click Continue.

• regedit

Navigate to: • HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\NlaSvc\Parameters\Internet

Under the Internet key, double-click EnableActiveProbing, and then in Value data, type: 0

The default for this value is 1. Setting the value to 0 prevents NCSI from connecting to a site on the Internet during checks for connectivity.

Click OK.

Restart the computer.

15


ARP Since ARP is required by IP for MAC address resolution, you have an opportunity to

measure ARP response time as well.

16

In this trace, the ARP response took 1 ms to reply

This exercise is also helpful to understand the impact of Network Load Balancing, Global Load Balancing, and other configurations that modify MAC addresses

Also another good way to look for ARP spoofing, ARP flooding or other anomalies


TCP Since we are focusing on web browsing, I will cover some TCP basics

17

The first thing is to locate a TCP SYN and SYN, ACK packet Sometimes you may have to use a combination of IP and TCP port filter In this example, the TCP SYN ACK took 48ms

Other things worth noting:• Win= is that device TCP Receiver Window Size• WS=Windows Scaling Option to increase your TCP Window Size• MSS= is that device Maximum Segment Size• SACK_PERM=indicates Selective Acknowledgements are enabled

o RFC 1072, and more recently by RFC 2018

In this example:• 10.44.10.171 (Me) supports SACK, and 74.208.29.25 (server) doesn’t• We are both using the maximum MSS

http://tools.ietf.org/html/rfc2018

http://tools.ietf.org/html/rfc2018


HTTP HTTP currently has three versions:

• .9• 1.0• 1.1

Some benefits that 1.1 brought with it was• Pipelining: the ability to send multiple commands over a connection• KeepAlive: keeps the TCP connection open

I always set my browser for 1.1, even if the server, proxy or firewall doesn’t support it everything should still work fine. Give it a test.

18

Internet Explorer


HTTP Response Time

One way to calculate HTTP response time is to use a display filter http and measure the delta time

In some cases you can reference the HTTP ETAG to ensure it’s the proper response

The ETAG is one of several mechanisms that HTTP provides for web cache validation

In this example you can see that the client and server are using HTTP ver 1.1

We can also see that the server is an Apache server when we look into the packet

19

https://en.wikipedia.org/wiki/Web_cache


HTTP Response Time

When calculating delay, try to avoid filtering on just the application layer (HTTP)

In this example we see the GET command in packet 10

The server sends back a TCP ACK in packet 11 proving it received packet 10, 49 ms later

The server then sends the data in packet 12

This type of behavior may start pointing to an overloaded server and worth monitoring

You also need to determine if your internal proxy, Firewall or NAT device is sending these on the server’s behalf or if it really came from the webserver

20


HTTP Response Time – Wireshark Specific

Wireshark will note • what packet number the request or response• Calculated response time

You should always validate that the calculated times are the same as your manual calculations since everybody does it a bit differently

21


HTTP Agent Examples Identifies the Application that sent the packet

22


HTTP Reports - Wireshark

Wireshark also has some helpful HTTP reports

23


HTTP Commands

Method/Commands References

GET RFC 1945

POST RFC 1945

HEAD RFC 1945

LINK RFC 1945

DELETE RFC 1945

OPTIONS RFC 2068

PATCH RFC 2068

PUT RFC 1945

TRACE RFC 2068

UNLINK RFC 1945

24

More common

http://www.networksorcery.com/enp/rfc/rfc1945.txt











HTTP STATUS CODE Categories

Category Description

1yz Informational.

2yz Success.

3yz Redirection.

4yz Client error.

5yz Server error.

25


HTTP STATUS CODESCode Description References100 Continue. RFC 2616101 Switching protocols. RFC 2616102 Processing. RFC 2518200 Ok. 201 Created. 202 Accepted. 203 Non-authoritive information. 204 No content. 205 Reset content. 206 Partial content. 226 IM used. 300 Multiple choices. 301 Moved permanently. 302 Moved temporarily. 303 See other. 304 Not modified. 305 Use proxy. 400 Bad request. 401 Unauthorized. 402 Payment required. 403 Forbidden. 404 Not found. 405 Method not allowed. 406 Not acceptable. 407 Proxy authentication required.

26




HTTP STATUS CODESe Description References407 Proxy authentication required. 408 Request timeout. 409 Conflict. 410 Gone. 411 Length required. 412 Precondition failed. 413 Request entity too large. 414 Request URI too large. 415 Unsupported media type. 426 Upgrade Required. 427 428 Precondition Required. RFC 6585429 Too Many Requests. RFC 6585430 431 Request Header Fields Too Large. RFC 6585500 Internal server error. RFC 2616501 Not implemented. RFC 2616502 Bad gateway. RFC 2616503 Service unavailable. RFC 2616504 Gateway timeout. RFC 2616505 HTTP version not supported. RFC 2616

506Variant Also Negotiates (Experimental).

RFC 2295

507 Insufficient Storage. RFC 4918508 Loop Detected. RFC 5842509 510 Not Extended. RFC 2774511 Network Authentication Required. RFC 6585

27






HTTP Reports – Network Instruments Observer

NI Observer also has some helpful HTTP reports

28


HTTP Credentials When someone logs into a website, you should note the authentication/encryption Protocol Analyzers can decode Basic Authorization using a base64 encoded string

29

Observer

Wireshark


Sample Baseline Items To Document

Client IP address Client HTTP ver Server name Server HTTP ver HTTP Errors Total Bytes Number of DNS name lookups DNS Errors TCP response time to server DNS response time to server Max Packet Size

30


HTTP Baseline to www.cnn.com

TASK Results

Client IP address 10.44.10.171

Client HTTP ver 1.1

Client Browser IE 11

Server name www.cnn.com

Server HTTP ver 1.1

Server App IIS 8.5

Total Bytes 2.8 MB

Number of DNS name lookups 250

TCP response time to server 35 ms avg to CNN.com

DNS response time to server 40 ms avg

Number of other Server IP’s 86 – used http display filter and endpoint report

Server IP, number of Bytes, resp - top 5 23.235.46.185 , 33,824, 35 ms

209.148.204.49, 19,336, 43 ms

149.174.149.39, 15,885, 23 ms

64.12.249.201, 15,593, 64 ms

209.148.204.56, 14,896, 34 ms

31

http://www.cnn.com/


Graphically Show Results

32

23.2

35.4

6.18

5

64.1

2.24

9.20

1

72.2

1.91

.29

68.6

7.15

2.15

8

107.

22.1

90.6

4

209.

148.

204.

40

74.1

21.1

39.1

9

23.2

35.3

9.17

5

69.1

71.2

6.70

239.

255.

255.

250

209.

148.

204.

26

98.1

39.2

25.4

3

54.8

4.24

2.4

66.2

35.1

53.3

6

23.4

9.91

.228

23.2

1.23

7.17

6

23.4

9.90

.99

50.1

16.1

94.2

1

199.

16.1

56.1

1

23.1

95.2

12.2

02

74.1

17.1

99.1

02

173.

194.

43.8

9

192.

155.

195.

220

157.

166.

238.

142

209.

148.

204.

50

66.1

85.8

5.29

31.1

3.73

.10

5000

10000

15000

20000

25000

30000

35000

40000

CNN Baseline - IP servers by Bytes

Dependency Analysis:The number if devices required to ‘build’ your page will affect the total user experience!

You are only as fast as your slowest time!


Other HTTP Reporting Options You can use built in features that some web browsers provide

• Chrome Developer Tools

33

Internet Explorer Developer Tools


Inefficient MSS From A Firewall/NAT Device In this example the Firewall could not support the maximum MSS of 1460

We see the webserver, or proxy/firewall/NAT with a MSS of 512

When the users experienced a problem, we noticed the MSS would drop to 2 Bytes• Some users even reported their PC’s would reboot

34

© 2015, The Technology Firm www.thetechfirm.com

Baselining

Getting Started

HTTPThanks for Watching

Tony Fortunato, Sr Network Performance Specialist

www.thetechfirm.com