Troubleshooting Sitecore Performance

Monitoring & Analyzing Performance IssuesAugust 12, 2013 1

Please review

before viewing this presentationhttp://www.slideshare.net/cwnelson/sitecore-performance-tipshttp://www.slideshare.net/cwnelson/sitecore-mobile-performance-tips

2

Sitecore Enterprise Web CMS

• Popular .NET Web Content Management System for mid-to-large organizations• Provides friendly tools for non-technical Content

Authors and Marketers• Can be customized and extended to meet almost

any conceivable business need

3

Troubleshooting Performance

Steps to troubleshoot performance issues• Create baseline logs of performance metrics• Monitor system when it is offline or off hours• Monitor system during normal operation

• Regularly review performance logs• Become familiar with normal metrics for your system

• Investigate performance metrics that are not normal• Scrutinize counters in areas of unusual performance• Try to narrow the scope of performance problems

• Take action to remove and or resolve bottlenecks4

Performance Monitoring

Monitoring helps locate root causes of issues• Inadequate resources or resource misallocation• Non-optimum coding or use of a legacy API• Missing configuration settings or misconfiguration

Monitoring helps identify resource bottlenecks• Server hardware & operating systems• IIS & ASP.NET • Databases• Sitecore

Monitoring is a recurring process, not an event• Familiarity with the system is a key success factor

5

Why Monitoring is Important

Building business trust• Ensuring performance meets user expectations• Averting business-impacting performance issues

Accomplishing cost-effective site management• Efficiently diagnosing root causes of issues• Perceiving the corrective actions needed to fix issues• Addressing issues instead of upgrading hardware• Accurately budgeting for future growth needs

6

Windows Performance Monitor

Windows Performance Monitor (PerfMon)• Use it to display graphs for all performance logs• System, IIS, ASP.NET, SQL, Sitecore counters, etc.• Computer Mgmt > Performance > Monitoring Tools• Use <Ctrl-h> to toggle highlight of selected counter

7

Understanding Bottlenecks

Performance bottlenecks• Performance of system is limited by a single resource• Bottleneck points are limiting components of system

Bottleneck indicators• One or more resources are over-consumed• Resource requests exceed available resources• Resource requests must be queued until available

• One or more resources are under-consumed• More subtle type of bottleneck that is harder to locate• Resource requests are not at their expected levels• The bottleneck blocking requests is elsewhere

8

Troubleshooting Bottlenecks

Identifying bottlenecks• Create a baseline log when system is offline or at rest• Monitor the system to log the current behavior• Compare the current behavior with the baseline log

Investigating bottlenecks• Review system counters before Sitecore counters• Look for sustained behaviors, not peaks or spikes• Over-consumption or under-consumption of resources• Use broad counters first; % Disk Time (Reads & Writes)• Use detailed counters to investigate suspicious values;

Avg. Disk Bytes Read & Avg. Disk Read Queue Length9

Viewing Server Performance Logs

Inspect server performance logs for issues• Check for signs of memory leaks• Creeping increases in consumption indicate a leak

• Check for signs of bottlenecks• High resource consumption for prolonged periods• Related queue length above 2

• Check for non-optimum metrics• Errors are signs of code issues or misconfiguration• Cache misses are signs of configuration issues

10

Server Bottlenecks

Server Bottlenecks• One or more server resources are over consumed• The server contains one or more bottleneck points

11

Server Bottleneck Types

Memory Processor Disk Network

Server Performance

Troubleshooting server performance issues• Log performance objects that contain key counters • Set logging interval to 15 minutes• Log data over multiple weeks

• Periodically review logged data for key counters • Use detailed counters to investigate suspect counters• Key counters that show signs of a potential problem• Periodically review Sitecore logs for nested exceptions• Rectify issues found from monitoring & Sitecore logs• Periodically review caching statistics & tune caches 12

Key Server Objects & Counters

Memory• Available Bytes• Committed Bytes

Processor • % Processor Time

System• Processor Queue Length

13

Physical Disk• % Disk Time• Avg. Disk Queue Length

Network Interface• Bytes Total/sec• Current Bandwidth• Output Queue Length

Warning Signs for Server Counters

Use performance counters to check warning signs• Memory: Committed Bytes is creeping up long term• Processor: % Processor Time is above 75% long term

and System: Processor Queue Length is above 10• Physical Disk: % Disk Time is above 50% long term and

Avg. Disk Queue Length is above 2 • Network Interface: Bytes Total/sec shows utilization is

above 25% and Output Queue Length is above 214

Memory Performance Counters

Key memory performance counters• Available Bytes - memory available for system use or apps• Value should be at least 10% of physical memory

• Committed Bytes - memory with page file space reserved • Value should be less than 75% of physical memory• A slow rise could indicate a memory leak

• Pages/sec - rate paging is used to resolve hard page faults• Preferred range is 0 to 20. Investigate if over 80

• Pool Nonpaged Bytes - memory for objects not paged• A slow rise could indicate a memory leak

• Pool Paged Bytes - memory for objects paged to disk

15

Processor Performance Counters

Key processor performance counters• % Processor Time – ratio CPU ran non-idle threads • Value should be less than 75% • Check for high value & Processor Queue Length > 10

Key system performance counters• Context Switches/sec - rate CPUs switch threads• A thread is preempted by a higher priority ready thread• CPU switches between user-mode and kernel-mode• High number can indicate high contention in locks

• Processor Queue Length - # threads in the CPU queue• Divide this value by the number of processors• A sustained queue > 10 threads per CPU is a problem

16

Disk Performance Counters

Key physical disk performance counters• % Disk Time - ratio that disk was busy with requests• Value should be less than 50%

• Avg. Disk Queue Length - requests that were queued• Disk delays and bottlenecks are likely when value > 2

Key paging file performance counters*• % Usage - ratio of Page File in use• % Usage Peak - ratio of peak usage of Page File

* Paging File counters are also related to memory counters 17

Network Performance Counters

Key network interface performance counters• Bytes Total/sec - rate of bytes over NICs• Network utilization should be less than 25%

• Output Queue Length – packet queue length• Value should be less than 2

18

Process Performance Counters

Key Sitecore process performance counters • Private Bytes - amount of memory used by process • Memory that cannot be shared with other processes

• Virtual Bytes - virtual address space used by process• Page File Bytes - space process reserved in page file• IO Data Operations/sec - rate process is doing I/O• IO Other Operations/sec - rate of control functions • % Processor Time – ratio process handled instructions

19

ASP.NET Performance Counters

Key ASP.NET performance counters• Application Restarts - # restarts during server uptime• High number indicates problems are causing recycles

• Requests Queued - # requests waiting for service• Increase shows server is at limit of concurrent requests• Raise default of 5,000 in the machine.config file

• Worker Process Restarts - # worker process restarts• Shows unexpected failures and intentional recycles

20

ASP.NET Application Counters

Key ASP.NET Applications performance counters• Errors Total - # of errors executing HTTP requests• Investigate parser, compilation and run-time errors

• Pipeline Instance Count – max # concurrent requests • Shows requests that are being processed for the app• High number signifies the CPU is not well utilized by app

• Requests/sec - current throughput of application• Values should remain within range under constant load

21

.NET Performance Counters

Key .NET CLR Exceptions performance counters• # of Exceps Thrown - # of exceptions thrown• A high value indicates a code or configuration problem• Check Sitecore logs for related application errors

Key .NET CLR Memory performance counters• # Induced GC - # calls to the Garbage Collector• A high value indicates a code or configuration problem• Remove calls to GC.Collect and GC.GetTotalMemory

• % Time in GC - time spent since end of the last cycle• Investigate managed heap when value is 50% or higher 22

IIS Performance Counters

Key Web Service performance counters• The following values should be as high as possible• Bytes Total/sec - # bytes transferred by the service• Total Method Requests/sec - HTTP requests received• Current Connections - # connections to web service

Key Web Service Cache performance counters• The following values should be as high as possible• File Cache Hits % - ratio successful lookups since startup• Kernel:URI Cache Hits % - ratio of cache hits to requests

• The following values should be as low as possible• Kernel:URI Cache Flushes - # files removed from cache• Kernel:URI Cache Misses - # unsuccessful cache lookups

23

SQL Performance Counters

Key SQL Buffer Manager performance counter• Buffer Cache Hit Ratio - % of pages found in buffer• Page did not have to incur a read from disk• Value should be as high as possible

Key SQL Statistics performance counters• Batch Requests/Sec - # received SQL batch requests• Over 1000 batch requests/sec may indicate a problem

• SQL Compilations/Sec - # Transact-SQL compilations • Over 100 compilations/sec may indicate a problem 24

SQL Access Performance Counters

Key SQL Access Methods performance counter• Page Splits/sec – splits from overflowing index pages• An index or data page can split when it becomes full• Page splitting can cause excessive I/O• Increase fill factor to allocate more space

25

Improving SQL Performance

Configuring SQL Server for best performance• Set compatibility level to SQL Server 2008 (100)• Set Auto Close and Auto Shrink properties to false• Set Recovery Model to simple• Check for percentage of fragmentation on indexes• Run the Index Physical Statistics Standard Report• Run report against each of the Sitecore databases• Rebuild indexes based on recommended operations

• Verify a Maintenance Plan exists and runs regularly• Clean up History, PublishQueue & EventQueue tables• Use SQL script in CMS Performance Tuning Guide

26

Sitecore Performance Counters

Key Sitecore Caching performance counters• The following value should be as high as possible• Cache Hits - # of cache hits for all caches

• The following values should be as low as possible• Cache Clearings - times that a cache has been cleared• Cache Misses - # of cache misses for all caches

Key Sitecore Data performance counters• Data.PhysicalReads - # of read operations from DB• Data.PhysicalWrites - # of write operations to DB 27

Sitecore Counters Configuration

Sitecore Performance Counters Configuration• Counters.Enabled – enable/disable counters • Enable performance counters when monitoring• Disable performance counters when not monitoring

• Counters.InstanceName – counter names• Change setting when you monitor multiple web apps

• Counters.ResetOnStartup – resets values to zero (0)

28

The Sitecore Health Monitor

The Sitecore Health Monitor background task• Periodically writes counter values to Sitecore logs• Supports Windows, CLR and Sitecore counters• Web.config contains healthMonitor pipeline settings• Interval specified by HealthMonitorInterval setting• Users can add counters for LogCounterStatus method• instanceType must be specified for added counters

29

The Counter Dump Agent

The Sitecore Counter Dump Agent• Periodically collects data on all Sitecore counters• Writes data to files in XML format • Files contain Sitecore counter values for time ranges• The frequency & filename can be set in web.config <agent type="Sitecore.Tasks.CounterDumpAgent" method="Run" interval="01:00:00"> <DumpFile>$(dataFolder)/diagnostics/counters.{date}.{time}.{processid}.txt</DumpFile> </agent>

30

Sitecore TaskManager Module

Sitecore Task Manager and Performance Monitor• Shared source module available from Marketplace• A web-based tool to monitor background processes• Indexing, scheduled tasks, publishing, etc.

http://marketplace.sitecore.net/Modules/TaskManager

31

Load Testing

Load Testing helps find slow running code paths• Most efficient way to identify code bottlenecks• Run load tests without caching during development• Use ANTS to profile your solution while stress testing

Load Testing helps ensure a system will scale• Creates simulated server loads expected on live site• Anticipates performance with increased traffic loads• A series of test cases are used during the testing• Best to monitor on both the server and client sides

32

Load Testing Tools

Server-based load testing• JMeter is a free stress testing/monitoring tool• Useful for identifying bottlenecks during development• Available at http://jmeter.apache.org/

• Web Capacity Analysis Tool (WCAT) is a free tool• http://www.iis.net/1466/ItemPermalink.ashx

Web-based load testing• Load Tester LITE is provided at no cost• Load Tester PRO can be rented or permanent license• Amazon has rental fees for cloud-based load engines• Available at http://www.webperformance.com/

33

In Summary

• Performance monitoring is a process, not an event• Use PerfMon to log system & Sitecore counters• Log counters for system, IIS, ASP.NET, db, Sitecore, etc.• Monitor performance objects containing key counters• Investigate additional detailed counters as needed• Use Health Monitor to log counters to Sitecore logs• Use Task Manager to monitor background processes• Use load testing to find slow running code paths

34

Caveat Emptor

There are many ways to monitor Sitecore performance• Skilled administrators can provide other solutions• Review your monitoring needs with your Sitecore partner• Review proposed changes with all stakeholders• Implement new configurations on a test system firstThis presentation is intended to be academic• To enhance public knowledge of performance monitoring• To promote better understanding of monitoring SitecoreAuthor cannot guarantee accuracy or fitness for your site• Author makes no representations or warranties• Author disclaims any liability if you act on this information

35

Acknowledgements

Wikimedia Commons Image of ASP.NET Architecture Stack

Wikimedia Commons Image of Network Topology

Wikimedia Commons Image of Warning Sign

Download slide presentation to view URLs

36

More Topics from TIC

All Sitecore-related SlideShares from TIChttp://www.slideshare.net/cwnelson/presentations

How to Assure Success for a New Sitecore Site (full slide deck)http://www.slideshare.net/cwnelson/creating-new-sitecore-sites-full

How to Avoid Shooting Yourself in the SEOhttp://www.slideshare.net/cwnelson/sitecore-seo-tips

Implementing Routine Sitecore Maintenancehttp://www.slideshare.net/cwnelson/sitecore-maintenance-tips

Performance Tips for Sitecore Mobile Usershttp://www.slideshare.net/cwnelson/sitecore-mobile-performance-tips

Sitecore 7 – New Features for Usershttp://www.slideshare.net/cwnelson/sitecore-7-new-features

Sitecore Performance Tips for Visitor Engagementhttp://www.slideshare.net/cwnelson/sitecore-performance-tips

37

The Inside Corner, Inc.

• The Inside Corner, Inc. (TIC) is owned by Craig Nelson• TIC is a Sitecore partner & web consultancy• A U.S. point of contact for teams of Sitecore specialists • We create new Sitecore sites & enhance existing ones

38

Primary Developer - Craig Nelson

• Sitecore Specialist certified for Sitecore 7• 10 years of system administration experience• 10 years as a Microsoft Certified Trainer• 5 years of ASP.NET web development experience• 4 years of Sitecore development experience• 3 years as a Sitecore technical instructor• Sitecore architecture & configuration expert• Performance & SEO optimization specialist• Workflow configuration specialist 39

Get Help with your Sitecore Site

Follow TIC on LinkedIn• http://www.linkedin.com/company/the-inside-corner-inc.

Add TIC to your Circles on Google Plus• http://google.com/+Theinsidecorner

Follow TIC on Twitter • https://twitter.com/TheInsideCorner

Contact Craig Directly• Craig Nelson - Certified 7.0 Developer and Sitecore Partner• cwnelson[at]ticdevs.com• Mobile - 707.319.7446

Visit the TIC website• http://ticdevs.com

40

Bibliography

All about Sitecore performance counters — A real hero

Optimizing Performance in Sitecore

SDN - Working with Sitecore Performance Counters

Sitecore CMS 6.0 -6.5 Diagnostics Guide

Sitecore CMS 6.0 -6.6 Performance Tuning Guide

Troubleshooting Performance Problems in SQL 2008

Download slide presentation to view URLs

41

Thank you for taking the time to view this presentation on

Troubleshooting Sitecore Performance

Please share with others and add a comment if you think it is useful

ThanksAugust 12, 2013

42