Upload
manuel-longo
View
1.208
Download
4
Embed Size (px)
DESCRIPTION
Learn about the tools and techniques that Microsoft Premier Support engineers use to gather data to troubleshoot and resolve issues. This session includes an overview of the troubleshooting process used to complete a Root Cause Analysis, and a review and demo of the different set of tools available for different needs including: -- Diagnostic Logging -- Data Collection -- Data Analysis -- Debugging
Citation preview
1 | SharePoint Saturday Chicago 2013Twitter: @SPSChicago Hashtag #SPSChicago
Available tools/techniques used by Microsoft Premier Support Engineers.
Manuel LongoSenior Manager – SharePoint Consultant – Sogeti US
SharePoint Troubleshooting Tools and Techniques
2 | SharePoint Saturday Chicago 2013Twitter: @SPSChicago Hashtag #SPSChicago
Manuel Longo Over 13 years of experience on Microsoft
Products & Technologies. Microsoft SharePoint Premier Field
Engineer for over 5 years. Microsoft Technical Solutions Professional
for over 3 years. Microsoft Consulting Services Developer
for over 1 year. Developer Consultant since 1998.
3 | SharePoint Saturday Chicago 2013Twitter: @SPSChicago Hashtag #SPSChicago
Outline & Agenda Troubleshooting Overview SharePoint Interdependencies SharePoint Performance/Capacity Analysis Troubleshooting Tools
Diagnostics Logging Data Collection Data Analysis Debugging
4 | SharePoint Saturday Chicago 2013Twitter: @SPSChicago Hashtag #SPSChicago
Troubleshooting Overview
5 | SharePoint Saturday Chicago 2013Twitter: @SPSChicago Hashtag #SPSChicago
Troubleshooting Overview Understand the system
Purpose Configuration Interdependencies
What is the issue/problem? End user experience IT Pro experience
Narrow the scope Happen to everyone or just a few users? All clients? Only ones in remote site? What’s common when it happens? What’s not when the problem is
absent?
Reproduce in Test Environment as possible
6 | SharePoint Saturday Chicago 2013Twitter: @SPSChicago Hashtag #SPSChicago
Root Cause Analysis (RCA) Identify the factors that resulted in the
nature, the magnitude, the location, and the timing of the harmful outcomes (consequences) of one or more past events in order to identify what behaviors, actions, inactions, or conditions need to be changed to prevent recurrence of similar harmful outcomes and to identify the lessons to be learned to promote the achievement of better consequences. "Success" is defined as the near-certain
prevention of recurrence.
http://en.wikipedia.org/wiki/Root_cause_analysis
7 | SharePoint Saturday Chicago 2013Twitter: @SPSChicago Hashtag #SPSChicago
Root Cause Analysis (RCA)1. Define the problem.2. Gather data and evidence.3. Identify the causes.4. Classify causes into causal factors
and root causes.5. Identify harmful factors.6. Identify corrective actions.7. Identify solutions.8. Implement recommended root
cause corrections.9. Ensure effectiveness.
Gather
Analyze
Correct
8 | SharePoint Saturday Chicago 2013Twitter: @SPSChicago Hashtag #SPSChicago
Diagnosing What is Changed“it was working 2 days ago, and suddenly started failing…”
…if it was working before, something must have changed.
Four questions to ask:1. What’s changed?2. What’s changed?3. No really…what’s changed?4. Something must have changed, what is it ?!?
9 | SharePoint Saturday Chicago 2013Twitter: @SPSChicago Hashtag #SPSChicago
SharePoint Interdependencies
10 | SharePoint Saturday Chicago 2013
Twitter: @SPSChicago Hashtag #SPSChicago
SharePoint Interdependencies SQL Server Active Directory (Kerberos) Name Resolution (DNS) Network IIS Service Applications Client side
HTTP/HTML/JavaScript
11 | SharePoint Saturday Chicago 2013
Twitter: @SPSChicago Hashtag #SPSChicago
SharePoint Performance/Capacity
12 | SharePoint Saturday Chicago 2013
Twitter: @SPSChicago Hashtag #SPSChicago
Low Performance/CapacityCheck SharePoint System Requirements Hardware and Software Browser Support IP Support Software Boundaries and limits
Boundaries vs. thresholds vs. limits Limits by hierarchy, by feature
Capacity Management vs. Capacity Planning Performance and Capacity Test Results Storage and SQL Server Capacity
Planning Monitoring!
http://technet.microsoft.com/en-us/library/cc262485.aspx
13 | SharePoint Saturday Chicago 2013
Twitter: @SPSChicago Hashtag #SPSChicago
Troubleshooting Tools
14 | SharePoint Saturday Chicago 2013
Twitter: @SPSChicago Hashtag #SPSChicago
Troubleshooting Tools Diagnostics Logging Tools
Event logs, IIS logs, Setup logs*, Upgrade logs*, Configuration logs*, ULS logs*, SharePoint Logging Database*.
Data Collection Tools SPSFarmReport*, Process Monitor, Process Explorer, Performance
Monitor.
Data Analysis Tools Log Parser, PAL, Fiddler,, Network Monitor/Wireshark, SQL profiler,
SQLDiag, Kerberos Tools, SPDiag*, ULS Viewer*.
Debugging Tools MIISClient , SPDisposeCheck*, MSOCAF, Developer Dashboard*,
IFilter View*, Debug Diag, WinDbg.
* - SharePoint specific.
15 | SharePoint Saturday Chicago 2013
Twitter: @SPSChicago Hashtag #SPSChicago
Diagnostics Logging Tools Event logs IIS logs Setup logs* Upgrade logs* Configuration logs* ULS logs* SharePoint Logging Database*
16 | SharePoint Saturday Chicago 2013
Twitter: @SPSChicago Hashtag #SPSChicago
Event Logs Have the same
purpose as any operating system or application logs.
Help predict and identify the sources of a problem.
Can help diagnose problems with SharePoint.
An error from the SharePoint Health Analyzer showing the need for an update
17 | SharePoint Saturday Chicago 2013
Twitter: @SPSChicago Hashtag #SPSChicago
Finding the issue…(which one?!)
Keep environment clean
18 | SharePoint Saturday Chicago 2013
Twitter: @SPSChicago Hashtag #SPSChicago
IIS Log Files Record data for a SharePoint web server in
various fields, such as Client IP Address, Server IP Address, Method, HTTP Status, Cookie, Protocol Substatus, etc. Contain a hyphen (-) in fields with no values and a plus sign (+) in fields with nonprintable characters.
Can be analyzed using Log Parser.
19 | SharePoint Saturday Chicago 2013
Twitter: @SPSChicago Hashtag #SPSChicago
IIS Log Configuration
Configuring IIS Logs to include additional data columns not included by default and how identify logs for particular web sites.
Demo
20 | SharePoint Saturday Chicago 2013
Twitter: @SPSChicago Hashtag #SPSChicago
SharePoint Setup Logs SharePoint setup log
Helps examine setup failures. Contains the list and status of the MSI packages
installed for SharePoint 2010. Stored in the temp directory of the user who initiated
setup. Foundation log named: Microsoft SharePoint Foundation
2010 Setup (YYYYMMDD-HHMMSS-SSS).log Server log named: SharePoint Server
Setup(YYYYMMDD-HHMMSS-SSS).log Uninstall log
Help determine if the MSI packages chained by the catalyst are uninstalled completely.
22 | SharePoint Saturday Chicago 2013
Twitter: @SPSChicago Hashtag #SPSChicago
SharePoint Configuration Logs Generated by the post-setup configuration tool,
PSConfig.exe and its GUI based tool, “SharePoint Configuration Wizard”.
Are created in the %CommonProgramFiles%\Microsoft Shared\web server extensions\14\LOGS folder.
Contain errors that occur during the post-setup configuration process.
Are easiest to review when imported to Microsoft Excel or another third-party log file analyzer.
Indicate whether each entry is of information or error type. Can be referenced, along with event logs, to obtain more
information about post-configuration errors displayed on the screen.
23 | SharePoint Saturday Chicago 2013
Twitter: @SPSChicago Hashtag #SPSChicago
SharePoint Configuration Logs Error on the
screen.
Error details in the configuration log.
Error details in the event log.
Begin trace logging for SharePoint 2010 Products Configuration Wizard. Version 14.0.4762.100007/10/2010 12:51:44 1 INF Entering function PsconfigUserInterfaceMain.Main07/10/2010 12:51:44 1 INF Entering function Common.SetCurrentThreadCultureToInstalledCulture07/10/2010 12:51:44 1 INF Entering function Common.SetThreadCultureToInstalledCulture07/10/2010 12:51:44 1 INF Current thread culture is English (United States), current thread ui culture is English (United States), installed culture is English (United States)07/10/2010 12:51:44 1 INF Leaving function Common.SetThreadCultureToInstalledCulture
24 | SharePoint Saturday Chicago 2013
Twitter: @SPSChicago Hashtag #SPSChicago
SharePoint Unified Logging System Are also referred to as trace logs in the
SharePoint GUI. Must have consistent location across all servers in
a farm. Contain over 266 categories of events. Contain nine columns: Timestamp, Process, TID,
Area, Category, EventID, Level, Message, Correlation.
25 | SharePoint Saturday Chicago 2013
Twitter: @SPSChicago Hashtag #SPSChicago
ULS Logging – Best Practices Change the drive to which the logs write. Restrict log disk space usage. Use the Verbose setting sparingly. Backup the logs regularly. Enable Event Log Flood Protection.
Use ULSViewer (demo later on in this module)
26 | SharePoint Saturday Chicago 2013
Twitter: @SPSChicago Hashtag #SPSChicago
SharePoint Logging Database Stores all SharePoint usage and health data, such as ULS
trace log data, event log data, blocking SQL queries, crawl and query statistics, feature usage, page requests, etc.
Is part of the Usage and Health Data Collection service application.
Has a public documented schema. Can be queried directly or written to by third-party
applications. Allows creating custom logging and diagnostic
providers. Has a customizable retention policy.
Retains detailed data for 14 days by default. Maximum 31 days of retention.
Configure it before you create other service applications. Otherwise it will be created automatically.
27 | SharePoint Saturday Chicago 2013
Twitter: @SPSChicago Hashtag #SPSChicago
Logging Database - Schema Logging database is tuned to support a heavy
load of simultaneous writes Up to 5000 transactions/second in parallel Should be put on a separate disk, if possible
The database can be used in various scenarios, including: Poor crawl or query performance SQL queries that are causing blocking Determining how widely a feature is actually
used Basic configuration is divided into two categories
—usage data and health data
28 | SharePoint Saturday Chicago 2013
Twitter: @SPSChicago Hashtag #SPSChicago
Logging Database - Reporting Table partitioning makes it easier to
use the provided views or create custom queries to view data in the logging database.
Most tables are partitioned for performance reasons.
Views are provided for usability. Excel can be used with the Excel Web
App or Web parts to create quick custom reports.
Some out-of-the-box reports make use of the logging database (next section).
29 | SharePoint Saturday Chicago 2013
Twitter: @SPSChicago Hashtag #SPSChicago
SharePoint Correlation IDs GUID identifiers that are internally associated with every
request, and are displayed with error messages. Using Windows PowerShell administrators can track down
the correlation ID in the ULS log and discovers the cause of the error.
Correlations IDs can be found in all the different Diagnostic Logs including: Event Logs. ULS Logs. SharePoint Logging Database. SQL Server (via Profiler) under events RPC:Starting and
SP:Starting events.
30 | SharePoint Saturday Chicago 2013
Twitter: @SPSChicago Hashtag #SPSChicago
Data Collection Tools SPSFarmReport* Process Monitor Process Explorer Performance Monitor
31 | SharePoint Saturday Chicago 2013
Twitter: @SPSChicago Hashtag #SPSChicago
SPSFarmReport Executable that lists all farm configuration
settings. Get it from CodePlex:
spsfarmreport.codeplex.com Available for MOSS 2007 and SharePoint 2010. Reports are best viewed in IE, as you can show
and hide each section (if you enable scripts) Useful for creating a baseline,
and then running periodically to check what may have changed.
32 | SharePoint Saturday Chicago 2013
Twitter: @SPSChicago Hashtag #SPSChicago
Process Monitor Combination of old Filemon and Regmon
utility. Catalogs interaction between the file
system and registry with filtering.
34 | SharePoint Saturday Chicago 2013
Twitter: @SPSChicago Hashtag #SPSChicago
Process Explorer Useful for understanding what is going on inside the
process. Shows currently active processes in the top windows. Bottom window shows either:
Handles for the selected process. DLLs and memory mapped files opened by the process.
35 | SharePoint Saturday Chicago 2013
Twitter: @SPSChicago Hashtag #SPSChicago
IIS Failed Request Tracing XML based files stored by default in \
inetpub\logs\FailedReqLogFiles\<W3SVC instance> Enabling consists of two steps:1. Enable Failed
Request Tracing.2. Configure your failure
definitions.
36 | SharePoint Saturday Chicago 2013
Twitter: @SPSChicago Hashtag #SPSChicago
Configuring IIS FREB
How to enable Failed-Request Tracing for the Site and configure the Log File Directory and failure definitions.
Demo
37 | SharePoint Saturday Chicago 2013
Twitter: @SPSChicago Hashtag #SPSChicago
Performance Monitor Captures data from the local machine or a remote
machine on different performance counters in either real time or regularly scheduled intervals.
Data can be stored and read in comma separated, tab separated, SQL, or binary format.
Common Counters: CPU. Disk. Memory. Network Interface.
38 | SharePoint Saturday Chicago 2013
Twitter: @SPSChicago Hashtag #SPSChicago
Data Analysis Tools Log Parser Performance Analysis of Logs (PAL) Fiddler Network Monitor or Wireshark SQL profiler SQLDiag Kerberos Tools SPDiag* ULS Viewer*
39 | SharePoint Saturday Chicago 2013
Twitter: @SPSChicago Hashtag #SPSChicago
Log Parser Provides universal query access to txt
based data sources; such as log files, XML files, and CSV files.
It was intended for use with the Windows operating system.
Free download from downloads.microsoft.com and was included with the IIS 6.0 Resource Kit tools.$ logparser <options> <SQL expression> $ logparser <options> <SQL expression>
$ logparser -e:IISW3C -q "SELECT date, time, cs-username FROM *.log WHERE cs-uri-stem LIKE '%.aspx' ORDER BY date, time;"
$ logparser -e:IISW3C -q "SELECT date, time, cs-username FROM *.log WHERE cs-uri-stem LIKE '%.aspx' ORDER BY date, time;"
40 | SharePoint Saturday Chicago 2013
Twitter: @SPSChicago Hashtag #SPSChicago
Logs parsing with LogParser
Use LogParser to query IIS and Diagnostic logs.
Demo
41 | SharePoint Saturday Chicago 2013
Twitter: @SPSChicago Hashtag #SPSChicago
Performance Analysis of Logs Wizard-based tool that generates a Windows
PowerShell script to analyze perfmon logs against best practice thresholds.
Thresholds are included for major Microsoft products like IIS, SharePoint, SQL Server, Exchange, etc.
Some thresholds change based on user entered criteria or computer role.
Results are created as a HTML based report.
42 | SharePoint Saturday Chicago 2013
Twitter: @SPSChicago Hashtag #SPSChicago
PAL - Reports
43 | SharePoint Saturday Chicago 2013
Twitter: @SPSChicago Hashtag #SPSChicago
Performance Analysis with PAL
Use PAL to process performance monitor logs, scheduling the report generation.
Demo
44 | SharePoint Saturday Chicago 2013
Twitter: @SPSChicago Hashtag #SPSChicago
Fiddler A GUI-based tool to examine HTTP traffic
to a browser. Looks at additional statistic information.
46 | SharePoint Saturday Chicago 2013
Twitter: @SPSChicago Hashtag #SPSChicago
Analysis of HTTP Traffic with Fiddler
Analysis of HTTP traffic opening Central Administration and SharePoint Sites and the effect of caching.
Demo
47 | SharePoint Saturday Chicago 2013
Twitter: @SPSChicago Hashtag #SPSChicago
Network Monitor (or Wireshark) Network packet sniffing tool. Free to download from
downloads.microsoft.com Many features for filtering, and adding rules to
make displaying data clear. Process tracking. Grouping by network
conversation. Support for over 300 public
and Microsoft proprietary protocols.
Simultaneous capture sessions.
Wireless Monitor Mode with supported wireless NICs.
Real-time capture and display of frames.
Reassembly of fragmented data.
Sniffing of promiscuous mode traffic.
Can read libpcap capture files. API to access capture and
parsing engine.
48 | SharePoint Saturday Chicago 2013
Twitter: @SPSChicago Hashtag #SPSChicago
SQL Server Profiler Using SQL Server Profiler, you can
do the following: Create a trace that is based on a
reusable template. Watch the trace results as the
trace runs. Store the trace results in a table. Start, stop, pause, and modify
the trace results as necessary. Replay the trace results. Correlate performance monitor
data. Analyze deadlocks.
Users must have the ALTER TRACE permission to run.
50 | SharePoint Saturday Chicago 2013
Twitter: @SPSChicago Hashtag #SPSChicago
SQL Diag SQLdiag can collect the following types of
diagnostic information: Windows performance logs. Windows event logs. SQL Server Profiler traces. SQL Server blocking information. SQL Server configuration
information.
52 | SharePoint Saturday Chicago 2013
Twitter: @SPSChicago Hashtag #SPSChicago
Kerberos Troubleshooting Examines common settings that
contribute towards Kerberos authentication and delegation DelegConfig
Verify kerberos tickets on a client machine KerbTray Klist (WS2008/R2)
SPN configuration SetSPN –x and -a
53 | SharePoint Saturday Chicago 2013
Twitter: @SPSChicago Hashtag #SPSChicago
SharePoint Diagnostic Studio Aggregates multiple sources of information
together about a farm for both collection and presentation of the data.
Can be installed on a farm server or remotely. Must have Farm Administrator rights. Projects are created within the tool to access the
data.
54 | SharePoint Saturday Chicago 2013
Twitter: @SPSChicago Hashtag #SPSChicago
ULS Viewer Parse and open multiple SharePoint ULS
logs (will concatenate them if you select multiple)
Reorder and resize columns, sort on any column.
Filter by Severity, Category, and Process or a custom text filter/search.
Group multi-line stack traces into single log entries for easy viewing and copy/paste.
View message easily in separate pane (No more scrolling in notepad).
55 | SharePoint Saturday Chicago 2013
Twitter: @SPSChicago Hashtag #SPSChicago
Analysis of SharePoint Logs with ULSViewer
Reviewing the filtering and highlighting features of the ULSViewer analyzing SharePoint Logs in real time.
Demo
56 | SharePoint Saturday Chicago 2013
Twitter: @SPSChicago Hashtag #SPSChicago
Debugging Tools MIISClient SPDisposeCheck* MSOCAF Developer Dashboard* IFilter View* Debug Diag WinDbg
57 | SharePoint Saturday Chicago 2013
Twitter: @SPSChicago Hashtag #SPSChicago
FIM User Interface (MIISClient) User Profile Imports are handled by Forefront Identity
Manager in the background. Central Administration’s user interface does not provide a
high level of detail during profile synchronization. Enter MIISClient
Located at: C:\Program Files\ Microsoft Office Servers\14.0\Synchronization Service\UIShell\miisclient.exe
Allows you to see what issues SharePoint may have when attempting to sync with Active Directory in real time.
Provides error codes and user friendly messages.
58 | SharePoint Saturday Chicago 2013
Twitter: @SPSChicago Hashtag #SPSChicago
SPDisposeCheck Command line tool that checks custom solutions
using the SharePoint object model to help ensure best practices and remove common causes of memory leaks.
It recursively searches and analyzes each managed module.
An updated set of “Do Not Dispose” rules have been added recently to improve the tool.
Can be integrated into Visual Studio. Add to the Tools menu Add SPDisposeCheck to your post build
commands.
59 | SharePoint Saturday Chicago 2013
Twitter: @SPSChicago Hashtag #SPSChicago
MSOCAF Microsoft SharePoint Online Code Analysis
Framework. Required to use before uploading solutions to
Office 365. Can be used on premise as well. Picks up where SPDisposeCheck leaves off. An Internet connection is required to run the tool
(Clickonce Installer)
60 | SharePoint Saturday Chicago 2013
Twitter: @SPSChicago Hashtag #SPSChicago
Developer Dashboard Helps perform advanced troubleshooting directly from the
page being rendered. Displays detailed diagnostics information, consolidated into
sections, for each page request. Helps test page customizations before deployment to
production. Is a farm-wide setting; turned off by default.
Can be set to on, off, or ondemand (2010 only). Can be configured via stsadm, which is easier than configuration with
Windows PowerShell:stsadm -o setproperty -pn developer-dashboard -pv On
Can be viewed by site collection administrators. Can be configured via Windows PowerShell for ondemand
usage.
61 | SharePoint Saturday Chicago 2013
Twitter: @SPSChicago Hashtag #SPSChicago
Developer Dashboard
How to turn on Development Dashboard and use it for request diagnosis and troubleshooting.
Demo
62 | SharePoint Saturday Chicago 2013
Twitter: @SPSChicago Hashtag #SPSChicago
iFilterView Use for troubleshooting search crawl issues
related to iFilters. It loads the iFilter for the file(s) with issues. It is also helpful for determining the names of
Crawled Properties returned by the iFilter. Download from
http://ifiltershop.com/downloads/IFilterView.zip
63 | SharePoint Saturday Chicago 2013
Twitter: @SPSChicago Hashtag #SPSChicago
Debug Diag GUI based tool for analyzing process dumps to
examine for cases of crashes and hangs. Contains:
Debugger Service. Debugger Host.
Built-in reports for memory pressure, crash/hang analysis, and SharePoint analysis.
64 | SharePoint Saturday Chicago 2013
Twitter: @SPSChicago Hashtag #SPSChicago
WinDbg Debugging tool used to capture and/or analyze
process dumps. A common tool for troubleshooting issues with
ASP.NET applications such as SharePoint.
65 | SharePoint Saturday Chicago 2013
Twitter: @SPSChicago Hashtag #SPSChicago
Lessons Learned (…sort of)
66 | SharePoint Saturday Chicago 2013
Twitter: @SPSChicago Hashtag #SPSChicago
YammerHow to Join the Conversation1. Download the Yammer phone app from the iTunes, Windows Phone, or
Android App Store.2. Join your company network, if you haven't already, at
http://www.yammer.com.3. Go to https://www.yammer.com/microsoftmidwestcustomers and click
"request an invitation". When you receive the invitation click "accept."4. Go to
https://www.yammer.com/microsoftmidwestcustomers/groups/spschicago and click "join." You will receive a welcome message in 24-72 hrs.
5. Once you have access you will be able to download posted presentations, ask questions of any of the subject matter experts and participate in any conversations going on in the group
67 | SharePoint Saturday Chicago 2013
Twitter: @SPSChicago Hashtag #SPSChicago
Housekeeping Please remember to submit your session
evaluation using our mobile application Follow SharePoint Saturday Chicago on
Twitter @spschicago & use hashtag #spschicago
68 | SharePoint Saturday Chicago 2013
Twitter: @SPSChicago Hashtag #SPSChicago
Thanks to Our Sponsors!