TMG Data Packager AnalysisPart1

By Suraj Singh Security Support Escalation Engineer

Microsoft Forefront EDGE team

Scope• Intro• Scenario• Analyzing individual logs collected by TMG data packager.• Correlating different logs to make sense and complete the picture and

understand what's going on.

INTRO

• TMGpackage.cab file is generated after we collect data using TMG data

packager tool while doing repro of the issue.• More info about TMG packager tool

http://blogs.technet.com/b/sooraj-sec/archive/2010/04/10/instructions-for-isa-data-packager-to-collect-data-in-repro-mode.aspx• This presentation talks about how to analyze the data collected by this

tool.• So first thing to do is unzip this cab file and proceed as explained in

coming slides.• Idea is to give starting point to begin data analysis by yourself.

Logs collected• It depends on the scenario and the issue and corresponding template

used on the TMG data packager wizard

• The logs collected also depends on the options we choose in the modify options

For Basic repro the logs would look like this after extracting the contents from the cab file.

BPA Reports

This report can be opened using ISABPA tool

What we get from BPA reports• Static configuration of the TMG• Build• EMS• Array• Monitoring logging and reporting.• Firewall policy• Networking(network configuration, ipconfig,routing table, NIC config, NLB

etc)• Services , processes.• Regkeys

TMGLogs

TMGLogs

Network Captures

ISAtracing

Tools• Netmon(to read network traces)• Excel(to read TMG logs)• Text reader/editor tools for searching the text.

Scenario• Issue : Outlook anywhere users disconnect intermittently and admin

sees, huge amount of failed connections in the TMG live logs with different status codes e.g. 10054,64,1460.• TroubleshootingFor status codes 10054 and 64 provided following blog post written by Thomas deztnerhttp://blogs.technet.com/b/isablog/archive/2007/06/25/rpc-over-http-logging-wildness.aspxBut there was no explanation for1460.

Data Analysis• In this case data was already collected by Admin as per instructions by

previous engineer, it was taken as Basic repro at the time of issue.• We used client ip of the client machine that was facing the issue to

filter the data.• From the nature of the issue, it was clear that four important logs

were going to help us i.e. TMG live logs, Network captures, ISA Tracing(this log can only be used by MS Engineers as it needs certain files to parse this log, I m putting this here for reference to my peers as well) ,ISA BPA(to look into config)as we needed to understand why issue was happening and these on the fly logs were over best bets.

Where to start with• We can choose BPA Reports- To understand the config, To know what

Ip ranges are in internal network, NIC configs and IP configs, NLB config , Obvious alerts.• Then in scenarios like this we can start with TMG logs, Admin

provided the client machine IP, so that was used to filter all the traffic that came from this client.• I have masked the client IP in the next snapshot but idea is that we

can use excel data filters to filter Live logs using client IP.

TMG Logs

After scrolling right on excel looked for filter info column and corresponding request id

ISATracing(for my MS peers only)• Used a text editor/reader tool to filter the ISAtracing log using the

request id found in the TMG Live Logs.• Filtered the log and located the result code 1460 in it as we can see in

the next slode.

Zoom in

Noise:WPPISAPUBLIC:(x.x.x.x:1234 ==> x.x.x.x:443) (x.x.x.x:56789 --- x.x.x.x:443), 0 bytes, "<NULL>", 1460(ERROR_TIMEOUT)

Network captures• Filter the network montor traces taken on the internal NIC with this combination

1.2.3.4:12345 --- 7.8.9.10:443(sample ip of TMG internal NIC and CAS server and sample ports as I m masking the original ip/ports)• With filter like Tcp.port==12345 and tcp.port ==443 as it makes a unique session, but could not find the traffic in the network capture as due to huge traffic netmon buffers ran out.From the error it was clear that there was time out and server did not respond properly and time out happened , since we could not get information about this unique session, next best plan was to see how good is the connectivity between TMG and the CAS server, so once again looked at the traces with a different display filter.

Conclusion and Action Plan• After looking at three logs in this sequence we were able to find why we were getting

1460 i.e. it was due to time out (ISAtracing) and then looking at the network traces we found that there was really bad connection between TMG and CAS server. To get the clear picture suggested Admin that from current data we were able to determine that there is really bad connectivity between TMG and CAS, we should also see the picture from CAS end to determine how it responds to traffic sent by TMG at the time of the issue.

• If the CAS server does not respond due to bad NIC/drivers etc or its because of a networking device between TMG and CAS. So suggested him to collect simultaneous data from CAS, TMG and Client at the time of the issue.

CAS : Network monitor capturesTMG server-TMG data packager with web proxy and publishing templateClient machine : Network monitor captures

Few Tricks with ISA Tracing using Text editor/reader tools apart from Request ID.• Keywords

Result code.Error i.e. Keyword ErrorHexadecimal error that you are getting while doing repro of the issue e.g.

0x8007003aIP address : port combination.Failed (keyword)Status code e.g. 1460

What to do if you see result of the keyword used• Its hard work converted to smart work• You need to locate your repro or issue traffic by finding IP address or port

or user name of the user who experienced the issue and once you are able to locate that note down the context id or Request id and then search with that context id in text analysis tool and you shall get one single conversation and then observe through this conversation if you notice issue in that.• You might have to repeat this if you saw multiple request id with your

initial search to locate the problem.• With practice this process becomes quicker so more you practice, better

and faster you become reading this log.

Q & A