Performance Analysis using Windows Performance ToolkitMichael MilirudProgram ManagerMicrosoft Corporation

CL16

Session Objectives And Takeaways

> Performance is critical> Live production system analysis > WPT Kit, Performance Analyzer tool

suite, and xperf tools> How to

> Collect a performance trace> Do basic analysis of a performance trace

Agenda

> Motivation> Overview of Windows Performance

Toolkit> Key Concepts> Case Studies> Conclusion

Why care?

> Motivation> Overview of Windows Performance

Toolkit> Key Concepts> Case Studies> Conclusion

Ensuring good performance

> Responsiveness> Unresponsive system Customer

dissatisfaction> Resource utilization

> CPU, disk, memory, network, battery> Resources are limited and shared,

use them effectively> Good citizenship

> It takes one component to impact system performance

Resource Usage Concepts

> Resource usage point of view> Machine constraints = reality

> Applications/services/drivers/OS compete for:> Processor time> Physical memory space> Virtual address space> Disk service time> Disk space> Network bandwidth> Battery power> …

> Machine constraints = resource usage limits

Performance Analyzer Preview

High level overview

What is going on behind the scenes?

demo

Performance Analyzer tool suite

> xperf: > Trace capture, processing, and command-line

analysis> xperfview:

> Visual trace analysis> Xbootmgr (and xbootmgrSleep):

> On/Off transition trace capture> hwpower2etw:

> TDMS-to-ETL converter> symcachegen:

> PDB-to-SymCache generator

Basic operations

> Trace capture> Graphs

> Frame List> Legend> Selection> Overlaying

> Summary tables> Gold bar> Column Chooser> Sorting

I Get It, But Where Do I Get It?

> Motivation> Overview of Windows

Performance Toolkit> Key Concepts> Case Studies> Conclusion

Windows Performance Toolkit

> Official kit> Built and used by the Windows organization

> Wide support range > Cross platform

> {Vista+} x {client, server}> Cross architecture

> Publicly available> Contains Performance Analyzer (WPA) tool

suite> WPA includes XPerf> Also includes GPUView and WpfPerf tools

> Strategic investment

Where is WPT Kit applied today?

> Bread and butter of Windows Engineering group

> Windows 7 performance work> Industry-wide OEM performance effort

What is Performance Analyzer?

> Tools for > Capture> Post-processing> Analysis

> Very mature> 10+ years in development > 4th generation

> Product quality code> Enterprise grade level> Core code reused inbox > Dev/Test/PM/UA – the whole deal > Documentation on MSDN

Why Use Performance Analyzer?

> Holistic performance analysis> All processes/threads> User + kernel mode> DPCs and ISRs> Scheduling> Disk and file I/O> Memory> Network

> Analyze what customer experiences> Catch the problem as it happens> Capture-Anywhere-Decode-Anywhere

> Very low overhead vs. alternatives> System-wide temporal “debugger”

Issues Performance Analyzer Can Help Root-cause

> Responsiveness issues> Long delays in applications> High CPU usage> High Disk usage> Slow On/Off transitions> Poor battery life> And much more!

The Big Picture

ETL file

ETW

Event Providers

Event Providers

Control/Status

Control/Status

ETW Session

2. Any component that has been instrumented with Event Tracing API

1. Collection of configurable in-memory buffers that is managed

by the kernel

3. Controls logging sessions

and enables/disables

providers

5. GUI trace analysis via graphs and summary

tables

6. CLI trace analysis via

actions

Data flow

XML file

Control/Status

XPerfView

XPerfSystem and

Symbol Information

Merged

ETL file

Control/Status

Post Processing

Action

4. Metadata injection

How Does Performance Analyzer Pull This Off?

> ETW is the magic behind the curtains> Event Tracing for Windows

> Core component since Windows 2000> High performance, low overhead,

highly scalable> ~2.5% CPU usage for sustained rate of

10,000 events/sec on a 2GHz CPU> For details, see “Event Tracing” on

MSDN

You Got My Attention – What’s Next?

> Motivation> Overview of Windows Performance

Toolkit> Key Concepts> Case Studies> Conclusion

Capture a Performance Trace

> A performance trace is a persisted form of a (complex) performance measurement

> …Perform the scenario in question…> xperf –stop perf!GeneralProfiles.InSequentialFile trace.etl

> xperf –start perf!GeneralProfiles.InSequentialFile

What is CPU Time Spent On?

xperf trace.etl

CPU Summary TableAggregated

columnsGrouping columns

CPU Summary Table

Now Show Me Some Action

> Motivation> Overview of Windows Performance

Toolkit> Key Concepts> Case Studies> Conclusion

Disk Usage

Marking 875 messages in Outlook 2007 folder as read is terribly slow

Case Study #1

Marking 875 messages in Outlook 2007

folder as read is terribly slow

Throughput can be cut by 90%

Marking 875 messages in Outlook 2007

folder as read is terribly slow

Disk Usage Study Summary

> Multiple components working well in isolation could impact performance when executed in parallel

> Fighting over disk dramatically impacts performance for all involved> Disk seeks can cut throughput by 90%

> Schedule all work that can wait as Idle tasks> See Task Scheduler documentation on

MSDN

Microsoft Confidential

Everything is slow

User complained that system was very slow across multiple scenarios including boot

Case Study #2

> User complained that system was very slow across multiple scenarios including boot

> User complained that system was very slow across multiple scenarios including boot

Case Study: Slow Disk

> Symptom: > Very slow system across multiple

scenarios, including boot> Key Findings:

> Very high disk utilization throughout the trace

> Disk was taking a long time to service I/Os

CPU Usage

User browsed E:\ (SD) drive from Explorer and opened a photo from the drive.Audio playback glitched.

Case Study #3

User browsed E:\ (SD) drive from Explorer and opened

a photo from the drive. Audio playback glitched.

CPU Case Study Summary

> One bad component can disrupt numerous good components

> High usage of any single resource can cause performance issues

> Spending too much CPU at DPC impacts performance

> Use DMA instead of PIO when transferring data

What Have We Learned?

> Motivation> Key Concepts> Overview of Windows Performance

Toolkit> Case Studies> Conclusion

Best Practices

> Resource efficiency> Responsiveness> Good Citizenship

Summary

> Good performance customer satisfaction

> WPT = official perf tools from/for Windows

> Performance Analyzer> xperf:

Trace capture, processing, and command-line analysis

> xperfview:Visual trace analysis

> xbootmgr:On/Off transition trace capture

Call to Action

> Download WPT Kit (see resources)> Capture traces across key scenarios> Evaluate resource utilization> Identify resource bottleneck> Invest in ETW instrumentation> Visit Windows Performance Analysis

Dev Center > Use the public forum

Resources

> Windows Performance Analysis Dev Center:> http://msdn.microsoft.com/en-us/perform

ance

> Event Tracing for Windows MSDN docs:> http://msdn2.microsoft.com/en-us/library/

aa363787.aspx> Windows Internals

> 5th edition is now out!

YOUR FEEDBACK IS IMPORTANT TO US!

Please fill out session evaluation

forms online atMicrosoftPDC.com

Learn More On Channel 9

> Expand your PDC experience through Channel 9

> Explore videos, hands-on labs, sample code and demos through the new Channel 9 training courses

channel9.msdn.com/learnBuilt by Developers for Developers….

© 2009 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

Addendum

Key new features since PDC’08

> WPT 4.5 (Win7 RC)> Generic Events support> Hierarchical legends> Configuration of symbol/symcache repository

from the GUI> Stack tracing support for manifest-based events> Hardware power analysis> Idle CPU analysis

Key new features since PDC’08

> WPT 4.6 (Win7 RTM)> Column reordering in summary tables via the column

chooser > Assisted resizing of graphs> Auto-snapping selection in check-point and count graphs> System Configuration dialog box tabs are now fully

featured summary tables> Flush visualization in the detailed disk I/O graph> Single pass multiple action execution> Auto-scaling of Y-axis in graphs based on presented data

series> Per-provider filtering in dumper> ReadyBoot analysis