Network Path andApplication Diagnostics
Matt MathisJohn HeffnerRagu Reddy
7/19/05
http://www.psc.edu/~mathis/papers/PathDiag20050719.ppt
Outline• What is the real problem?
– Lessons from Web100– A new perspective
• Application and upper layer diagnosis– LAN bench testing
• Path and lower layer diagnosis– The pathdiag tool– A diagnostic server
• Future work– Alpha deployment– Open research questions
TCP tuning requires expert knowledge• By design TCP/IP hides the ‘net from upper layers
– TCP/IP provides basic reliable data delivery– The “hour glass” between applications and networks
• This is a good thing, because it allows:– Invisible recovery from data loss, etc– Old applications to use new networks– New application to use old networks
• But then (nearly) all problems have the same symptom– Less than expected performance– The details are hidden from nearly everyone
TCP tuning is painful debugging• All problems reduce performance
– But the specific symptoms are hidden• Any one problem can prevent good performance
– Completely masking all other problems• Trying to fix the weakest link of an invisible chain
– General tendency is to guess and “fix” random parts– Repairs are sometimes “random walks”– Repair one problem at time at best
The Web100 project• When there is a problem, just ask TCP
– TCP has the ideal vantage point• In between the application and the network
– TCP already “measures” key network parameters• Round Trip Time (RTT), available data capacity, etc• Can add many more
– TCP can identify the bottleneck• Why did it stop sending data?
– TCP can even adjust itself• “autotuning” eliminates one major class of flaws
See: www.web100.org
The next step• Web100 tools still require too much expertise
– They are not really end user tools– Too easy to overlook problems– Current diagnostic procedures are still cumbersome
• New insight from web100 experience– Nearly all symptoms scale with round trip time
• New NSF funding– Network Path and Application Diagnosis (NPAD)– 3 Years, we are at the midpoint
Nearly all symptoms scale with RTT
• For example– TCP Buffer Space, Network loss and reordering, etc– On a short path TCP can compensate for the flaw
• Local Client to Server: all applications work– Including all standard diagnostics
• Remote Client to Server: all applications fail – Leading to faulty implication of other components
Examples of flaws that scale• Chatty application (e.g., 50 transactions per request)
– On 1ms LAN, this adds 50ms to user response time– On 100ms WAN, this adds 5s to user response time
• Fixed TCP socket buffer space (e.g., 32kBytes)– On a 1ms LAN, limit throughput to 200Mb/s– On a 100ms WAN, limit throughput to 2Mb/s
• Packet Loss (e.g., 0.1% loss at 1500 bytes)– On a 1ms LAN, models predict 300 Mb/s– On a 100ms WAN, models predict 3 Mb/s
The confounded problems• For nearly all network flaws
– The only symptom is reduced performance– But the reduction is scaled by RTT
• On short paths, most flaws are undetectable– False pass for even the best conventional diagnostics– Leads to faulty inductive reasoning about flaw locations– This is the essence of the “end-to-end” problem– Current state-of-the-art diagnosis relies on tomography
and complicated inference techniques
The solutions• New diagnostic techniques to compensate for
“symptom scaling” • For applications (and upper layers)
– Bench test over an (emulated) ideal long path• For path testing (and lower layers)
– Test path sections using a instrumented application that can extrapolate test results to a long path
Diagnosing applications• Goal: Tools to “bench test” applications in the lab
– Client and server on the same LAN• App developer has easy access to all components
– Emulate a long ideal path between client and server• Also checks some OS and TCP features• Several different techniques (next topic)
• Developer gets first hand experience with delay– If it fails in the lab, it will not work on a WAN– Can not blame the network– Can not repeal the speed of light– Has to fix the application
Emulating delay• Multiple techniques to emulate long paths
– Scenic routing via tunnels– Kernel delays (e.g. netem, nistnet, dummynet)– Application (pipe) delay via a proxy
• We have ~5 techniques prototyped/under test– Kernel hacking vs non-privileged users– Ease of use/ease of installation– Maximum data rate– Authenticity of the delay
The high end• GRE tunnel
– Encapsulate in either client or server– Decapsulate in a carrier grade router
• Can support authentic delays at rates up to 10 Gb/s– PSC uses this to test ETF apps and infrastructure
• Very expensive– Requires carrier grade gear– Additional “tunnel pic”– Awkward (non-scaleable) configuration management– Generally requires privileged access to critical HW
The low end• Simple user mode application proxy to forward connections
– Connect from client to proxy– Proxy connects to server– (Think “man in the middle” attack)
• Proxy copies application data between the two sockets– but with a scheduler and timers to delay the data– Implements pure “application delay” over ideal TCP
• Cheap and easy for low rate applications– Non-privileged “c” program (POSIX(?))– Can be controlled and configured with a web form
• But beware of the security implications
• Interacts badly with authentication, etc– All of the same problems as NAT
Starting to look at common applications• Classic ssh & scp
– Session setup takes many RTTsAuthentication, negotiating options & encryption, etc24 RTTs in one typical(?) example
– Internal flow control• Subject of Chris Rapier’s “HPN-SSH” talk this afternoon• (This work predates NPAD)
• Thinking about X11 and others
Testing the path• Need to test short sections to localize a flaw
– But “symptom scaling” normally hides a failing section• New tool (“pathdiag”):
– Measure the performance of each short section• Use Web100 to collect detailed statistics• Loss, delay, queuing properties, etc
– Use models to extrapolate results to the full path• Assume that the rest of the path is ideal• You have to specify the end-to-end performance goal
– Data rate and RTT
– Pass/Fail on the basis of the extrapolated performance
Deploy as a Diagnostic Server
• Use pathdiag in a Diagnostic Server (DS)• Specify End to End target performance
– from server (S) to client (C) (RTT and data rate)• Measure the performance from DS to C
– Use Web100 in the DS to collect detailed statistics– Extrapolate performance assuming ideal backbone
• Pass/Fail on the basis of extrapolated performance
Example 1- good news
Example 1, continued
Example 2 - not so good
Example 2, continued
Key pathdiag/DS features• Coverage for a majority of OS and network flaws
– Most of the remaining flaws can be detected with pathdiag in the client or traceroute
– Eliminates nearly all(?) false pass results• Tests becomes more sensitive on shorter paths
– Conventional diagnostics become less sensitive– Depending on models, perhaps too sensitive
• New problem is false fail (e.g. queue space tests)
• Flaws no longer completely mask other flaws– A single test often detects several flaws
• E.g. find both OS and network flaws in the same test
– They can be repaired concurrently
Key features, continued• Results are specific and less geeky
– Intended for end-users– Provides a list of action items to be corrected
• Failed tests are showstoppers for HPN apps– Details for escalation to network or system admins
• Archived DS results include raw web100 data– Can reprocess with updated reporting SW
• New reports from old data
– Critical feedback for the NPAD project• We really want to collect “interesting” failures
Blast from the past• Same base algorithm as “Windowed Ping” [Mathis, INET’94]
– Aka “mping”– See http://www.psc.edu/~mathis/wping/– Killer diagnostic in use at PSC in the early 90s– Stopped working with the advent of “fast path” routers
• Use a simple fixed window protocol– Scan window size in 1 second steps– Measure data rate, loss rate, RTT, etc as window changes
Plan for the Future• Application bench testing
– Considering running a tunnel service for SC2005• Contact us if this would be useful
– Will release the delay proxy software– Plan to release cookbooks for non-proxy methods
Using tunnels, netem, etc
• Pathdiag/DS – Hope to deploy across I2
• Co-locate with NDT servers• Collect interesting test results to refine the tool
– Release pathdiag and DS sources• Bridging the Gap workshop
– http://e2epi.internet2.edu/btg/