5/30/00 CSE 225

Performance Analysis Tools

Nadya Williams

Spring, 2000UCSD

5/30/00 CSE 225

Outline

• Background

• Performance measurement

• SvPablo

• Autopilot

• Paradyn

• XPVM

5/30/00 CSE 225

Background• Goal - high performance computing for

applications that are distributed:– by design, e.g. collaborative environments,

distributed data analysis, computer-enhanced instruments

– by implementation, e.g. metacomputing, high-throughput computing

• Goal - to achieve & maintain performance guarantees in heterogeneous, dynamic environments

5/30/00 CSE 225

Background

Performance-robust grid applications need to• Identify resources required to meet application

performance requirements• Select from problem specification, algorithm &

code variants• Establish hierarchical performance contracts• Select and manage adaptation strategies when

performance contracts are violated

5/30/00 CSE 225

VizEngine

VizEngine

Computational grids

Visualizationand Steering

MPPMPP

Real-timeData Analysis

NetworkNetwork

Network

Shared resources– computation, network, and data archives

5/30/00 CSE 225

Complexity

• Emerging applications are dynamic– time varying resource demands– time varying resource availability– heterogeneous execution environments– geographically distributed

• Display and analysis hierarchy– code, thread, process, processor– system and local area network– national/international network

5/30/00 CSE 225

Grid performance challenges

• Wide area infrastructure• Many resource models• Behavioral variability

– complex applications, diverse systems and networks

– irreproducible behavior

• Heterogeneous applications– multilingual and multimodel

– real-time constraints and shared resources

• Prediction & scheduling

5/30/00 CSE 225

Outline

• Background

• Performance measurement

• SvPablo

• Autopilot

• Paradyn

• XPVM

5/30/00 CSE 225

Performance analysis

• The ability to– capture– analyze– present– optimize

• Multiple analysis levels– hardware– system software– runtime systems– libraries– applications

Good tools must accommodate all

5/30/00 CSE 225

Real-time Multilevel Analysis

• Multilevel Drilldown– multiple sites– multiple metrics– real-time display

• Problems– uncertainty and perturbation– confusion of cause and effect

5/30/00 CSE 225

Guidelines

• Design for locality– regardless of programming model– threads, MPI, data parallel -- it’s the same

• Recognize historical models– large codes develop over time– assumptions change

• Think about more than FLOPS– I/O, memory, networking, user interfaces

5/30/00 CSE 225

Initial steps

• Develop infrastructure for structural and performance information

• Provide instrumentation of end-user applications & communication libraries

• Study performance characteristics of real grid applications

5/30/00 CSE 225

Peak and Sustained Performance

• Peak performance– perfect conditions

• Actual performance– considerably less

• Environment dictates performance– locality really matters– we must design for performance stability– more of less may be better than less of more

5/30/00 CSE 225

Measurement developments• Hardware counters

– once rare (Cray), now common (Sun, IBM, Intel, Compaq)

– metrics

• operation types

• memory stalls

• Object code patching – run-time instrumentation

• Compiler integration– inverse compiler transformations

– high-level language analysis

5/30/00 CSE 225

Correlating semantic levels

• Performance measurements – capture behavior of executing software– reflect output of multi-level transformations

• Performance tools– must relate data to “user” semantic model

• cache miss ratios cannot help a MATLAB user

• message counts cannot help an HPF user

– should suggest possible performance remedies

5/30/00 CSE 225

Analysis developments• Visualization techniques

– traces and statistics

• Search and destroy– AI suggestions and consultants– critical paths and zeroing

• Data reduction and processing– statistical clustering/projection pursuit– neural net, and time series classification

• Real-time control– sensor/actuator models

5/30/00 CSE 225

Performance tool checkpoint• An incomplete view …

– representative techniques and tools

• Major evolution– from architectural views/post-mortem

analysis – to deeper correlation and derived metrics

• Key open problems– adaptivity– scale– semantic correlation

5/30/00 CSE 225

Representative vendor tools• IBM VT™

• “ParaGraph” trace display and statistical metrics

• Silicon Graphics Speedshop™• R10000, R12000 hardware counter tools

• Pallas Vampir™• event tracing and display tools

• Cray ATExpert™ (autotasking)• basic AI suggestions for tuning

• Intel SPV™• ParaGraph and hardware counter displays

• TMC/SUN Prism™• data parallel and message passing analysis

5/30/00 CSE 225

Representative research tools• Illinois SvPablo™

– performance data metaformat– Globus integration (sensor/actuator control)

• Illinois Autopilot™– performance steering

• Wisconsin Paradyn™– runtime code patching– performance consultant

• Oak Ridge National Lab XPVM– X Windows based, graphical console and monitor for

PVM

5/30/00 CSE 225

Outline

• Background

• Performance measurement

• SvPablo

• Autopilot

• Paradyn

5/30/00 CSE 225

Department of Computer ScienceUniversity of Illinois at

Urbana-Champaign

SvPablo:

Graphical source code browser for performance tuning and

visualization

5/30/00 CSE 225

SvPablo Outline

• Background

• SvPablo overview

• SvPablo model

• Automatic/Interactive instrumentation of programs

• The Pablo Self-Defining Data Format

5/30/00 CSE 225

SvPablo Background• Motivations

– emerging high-level languages (HPF and HPC++)– aggressive code transformations for parallelism– large semantic gap between user and code

• Goals– relate dynamic performance data to source– hide semantic gap– generate instrumented executable/simulated code– support performance scalability predictions

5/30/00 CSE 225

Background

• Tools should provide the performance data and suggestions for performance improvements at the level of an abstract, high-level program

• Tools should integrate dynamic performance data with information recorded by the compiler that describes the mapping from the high-level source to the resulting low-level explicitly parallel code

5/30/00 CSE 225

SvPablo overview

A graphical user interface tool for:• source code instrumentation• browsing runtime performance dataTwo major components:• performance instrumentation libraries• performance analysis and presentationProvides:• performance data capture• analysis• presentation

5/30/00 CSE 225

SvPablo overview

• Instrumentation– automatic

• HPF (from PGI)

– interactive• ANSI C• Fortran 77• Fortran 90

• Data capture– dynamic software statistics (no traces)– SGI R10000 counter values

5/30/00 CSE 225

SvPablo overview• Source code instrumentation

– HPF: PGI runtime system invokes instrumentation

• each procedure call

• each HPF source line

– C and Fortran programs: interactively instrumented

• outer loops

• function calls

• Instrumentation maintains statistical summary• Summaries correlated across processors• Correlated summary input to browser

5/30/00 CSE 225

SvPablo overview• Architectures:

– any system with the PGI HPF compile– any system with F77 or F90– C applications supported on

• single processor Unix workstations• network of Unix workstations using MPI• Intel Paragon• Meiko CS2

• GUI supports:– Sun (Solaris)– SGI (IRIX)

5/30/00 CSE 225

Statistics metricsFor procedures :

– count– exclusive / inclusive duration– send / receive message duration (HPF only)

For lines:– count– duration– exclusive duration– message send and message receive (HPF only)

• duration• count• size

– event counters (SGI)

Mean, STD, Min, Max

5/30/00 CSE 225

SvPablo model

. . .

. . .

5/30/00 CSE 225

New project dialog box

5/30/00 CSE 225

HPF performance analysis data flow

SvPabloCombine

Par

alle

lA

rch

itec

ture

SvPablodata capture

library

Per-processperformance

files

HPFsource code

instrumentedobject code

Linker

instrumentedexecutable

PGI HPF

compiler

performancefile

Graphical performance

browser

5/30/00 CSE 225

HPF instrumentation

• pghpf -c -Mprof=lines source1.F

• pghpf -c -Mprof=lines source2.F

• pghpf -Mstats -o prog source1.o source2.o

/usr/local/SvPablo/lib/pghpf2SDDF.o

• prog -pghpf -np 8

• SvPabloCombine HPF_SDDF*

5/30/00 CSE 225

Performance visualization

Metrics:count &

exclusiveduration

5/30/00 CSE 225

Performance metric selection dialog

5/30/00 CSE 225

C / F77/ F90 data flow

Par

alle

lA

rch

itec

ture

SvPablodata capture

library

SvPabloCombine

per-processperformance

files

performancefile

instrumentedsource code

instrumentedobject code

Linker

instrumentedexecutable

compiler

create or editproject

visualizeperformance

file

InstrumentC or Fortran

files

5/30/00 CSE 225

Interactive instrumentation

InstrumentableConstructs

(function callsand outer loops)

5/30/00 CSE 225

Generating an instrumented executable program

• mpicc -c file1.Context1.inst.c

• mpicc -c file2.Context1.inst.c

• mpicc -c Context1/InstrumentationInit.c

• mpicc -o instFile InstrumentationInit.o

file1.Context1.inst.o

file2.Context1.inst.o

svPabloLib.a

5/30/00 CSE 225

SDDF: a medium of exchange

Self-Defining Data Format– data meta-format language for performance

data description– specifies both data record structures and data

record instances– separates data structure and semantics– allows the definition of records containing

scalars and arrays– supported by the Pablo SDDF library

5/30/00 CSE 225

SDDF files: classes of records

• Command: conveys action to be taken

• Stream Attribute: gives information pertinent to the entire file

• Record Descriptor: declares record structure

• Record Data: encapsulates data values

5/30/00 CSE 225

Record descriptorsDescribe record layout

• Each Record Descriptor contains:• A unique tag and record name

• An optional Record Attribute

• Field Descriptors, each one containing:• an optional Field Attribute • field type specifier• field name• optional field dimension

5/30/00 CSE 225

SDDF: record descriptor & data#300:// "description" "PGI Line-Based Profile Record""PGI Line Profile" { int "Line Number"; int "Processor Number“[]; int "Procedure ID"; int "Count"; double "Inclusive Seconds"; double "Exclusive Seconds"; int "Send Data Count"; int "Send Data Byte"; double "Send Data Seconds"; int "Receive Data Count"; int "Receive Data Byte"; double "Receive Data Seconds";}; "PGI Line Profile" {359, [2]{7,9}, 4, 399384, 31.071, 31.071, 0, 0, 0, 0, 0,

0};;

tag

record name

field descriptors

5/30/00 CSE 225

SvPablo language transparency

Meta-format for performance data• language defined by line and byte offsets

• metrics defined by mapping to offsets

• SDDF records• performance mapping information

• performance measurements

Result• language independent performance browser

• mechanism for scalability model integration

5/30/00 CSE 225

SvPablo conclusionsVersatility – yes• analysis GUI is quite versatile, provides the ability to

define new modules, but steep learning curve • theoretically, any type of view could be constructed from

the toolkit provided

Portability – not quite• Intended for wide range of parallel platforms and

programming languages, reality is different – (SUN, SGI)

Scalability - some• Pablo trace library monitors and dynamically alters the

volume, frequency, and types of event data recorded• not clear how: automatically or by user at low level?• need to integrate predictions

5/30/00 CSE 225

Outline

• Background

• Performance measurement

• SvPablo

• Autopilot

• Paradyn

• XPVM

5/30/00 CSE 225

Autopilot - a performance steering toolkit

Provides flexible infrastructure for real-time adaptive control of parallel and distributed computing resources

Department of Computer ScienceUniversity of Illinois at

Urbana-Champaign

5/30/00 CSE 225

Autopilot outline

• Background

• Autopilot overview

• Autopilot components

• Conclusions

5/30/00 CSE 225

Autopilot background• HPC: from single parallel systems to distributed

collections of heterogeneous sequential and parallel systems.

• emerging applications are irregular– have complex, data dependent execution behavior

– dynamic, with time varying resource demands

• failure to recognize that resource allocation and management must evolve with applications

Consequence: small changes in application structure

can lead to large changes in observed performance.

5/30/00 CSE 225

Autopilot background

• interactions between application and system resources change – across applications

– during a single application's execution

Autopilot approach : create adaptable– runtime libraries

– resource management policies

5/30/00 CSE 225

Autopilot overviewAfter the integration of

– dynamic performance instrumentation – on-the-fly performance data reduction – configurable, malleable resource management algorithms – real-time adaptive control mechanism

Have adaptive resource management infrastructure Given:

– application request patterns– observed system performance

Automatically choose & configure resource management algorithms:– increase portability – increase achieved performance

5/30/00 CSE 225

Autopilot components

1. Autopilot - implements the core features of the Autopilot system.

2. Fuzzy Library - needed to build the classes supporting the fuzzy logic decision procedure infrastructure

3. Autodriver - provides a graphical user interface (written in Java)

4. Performance Monitor - provides tools to retrieve and

record various system performance statistics on a set of machines.

5/30/00 CSE 225

1 Autopilot component

• libAutopilot.a – creation, registration, and use – sensors

– actuators (enable and configure resource management policies)

– decision procedures

• AutopilotManager - a utility program which displays the sensors and actuators currently registered with the Autopilot Manager

5/30/00 CSE 225

2 Fuzzy library component

• Fuzzy Rules to C++ translator

• related classes used by the Autopilot fuzzy logic decision procedure infrastructure.

5/30/00 CSE 225

3 Autodriver component

• Autopilot Adapter program– provides a Java interface to Autopilot (must run on UNIX)

• JAVA GUI – talks to Autopilot through the Adapter– allows a user to monitor and interact with live

sensors and actuators. (runs on any platform that supports Java)

5/30/00 CSE 225

4 Performance monitor component

two kinds of processes• Collectors

– run on the machines to be monitored – capture quantitative application and system

performance data

• Recorders – compute performance metrics. – record or output it.

communicate via Autopilot component

5/30/00 CSE 225

Closed loop adaptive control

Knowledge Repository

Fuzzy SetsRules

Fuzzy LogicDecision Process

Fuzz

ifier

Def

uzzi

fier

Inp

uts

Ou

tpu

ts

System

SensorsSensors

ActuatorsActuators

Illinois Autopilot Toolkit Illinois Autopilot Toolkit (Reed (Reed et alet al))

Real-time measurementGlobus integration

5/30/00 CSE 225

Autopilot conclusions

• Goal is creation of an infrastructure for building resilient, distributed and parallel applications.

• allow the creation of software that can change its behavior and optimize its performance in response to real-time data – on software dynamics

– performance.

• order of magnitude performance improvements

5/30/00 CSE 225

Outline

• Background

• Performance measurement

• SvPablo

• Autopilot

• Paradyn

5/30/00 CSE 225

Paradyn

performance measurement tool for parallel and distributed programs

Computer Science,

University of Wisconsin

5/30/00 CSE 225

Paradyn outline

• Motivations

• Approach

• Performance Consultant

• Conclusions

5/30/00 CSE 225

Paradyn motivations

• provide a performance measurement tool that scales to long-running programs on large parallel and distributed systems

• automate much of the search for performance bottlenecks

• avoid the space and time overhead typically associated with trace-based tools.

• go beyond post-mortem analysis

5/30/00 CSE 225

Paradyn approach

Dynamic instrumentation – based on dynamically controlling what performance

data is to be collected.

– allows data collection instructions to be inserted into an application program during runtime.

Paradyn• dynamically instruments the application• automatically controls the instrumentation in

search of performance problems

5/30/00 CSE 225

Paradyn model

• the Paradyn front-end and user interface– display performance visualizations

– use the Performance Consultant to find bottlenecks

– start and stop the application

– monitor the status of the application

• the Paradyn daemons – monitor and instrument the application processes.

5/30/00 CSE 225

Performance consultant module

• automatically directs the placement of instrumentation

• has a knowledge base of performance bottlenecks and program structure

• can associate bottlenecks with specific causes and with specific parts of a program.

5/30/00 CSE 225

Paradyn runtimeConcepts for performance data analysis/presentation1. metric-focus grid – cross-product of two vectors

– list of performance metrics (CPU time, blocking time…)– list of program components (procedures, processors, disks)– elements of the matrix can be single-valued (e.g., current – value, average, min, or max) or time-histograms

2. time-histogram – fixed-size data structure recording behavior of a metric as it varies over time

Performance data granularity– global phase– local phase

5/30/00 CSE 225

Performance consultantWisconsin Paradyn Toolkit (Miller Wisconsin Paradyn Toolkit (Miller et alet al))

unknowntrue false

5/30/00 CSE 225

Performance consultantWisconsin Paradyn Toolkit (Miller Wisconsin Paradyn Toolkit (Miller et alet al))

5/30/00 CSE 225

Outline

• Background

• Performance measurement

• SvPablo

• Autopilot

• Paradyn

• XPVM

5/30/00 CSE 225

XPVM

Graphical console and monitor for PVM

developed at the Oak Ridge National Lab

• Provides a graphical user interface to the PVM console commands

• Provides several animated views to monitor the execution of PVM programs

5/30/00 CSE 225

XPVM overview

• Xpvm generates trace records during PVM program execution. The resulting trace file is used to "playback" a program's execution.

• The xpvm views provide information about the interactions among tasks in a parallel PVM program, to assist in debugging and performance tuning.

• Xpvm writes a Pablo self-defining trace file

5/30/00 CSE 225

XPVM menus

• Host menu permits to configure a parallel virtual machine by adding/removing hosts

• Tasks menu enables to spawn, signal, or kill PVM processes, can monitor selected PVM system tasks, such as the group server process

5/30/00 CSE 225

XPVM menus

• Reset menu resets parallel virtual machine, xpvm views, or trace file

• Help menu provides help features

• Views permits selection of any of the five xpvm displays for monitoring program execution

5/30/00 CSE 225

XPVM menus

• Trace file play back controls - play, step forward, stop or reset the execution trace file

• Trace file selection window - displays the name of the current trace file

5/30/00 CSE 225

XPVM views (5)

Network• Displays high-level activity on each node in the

virtual machine • Each host is represented by an icon image

showing host name and architecture • Icons are color illuminated to indicate status

– Active - at least one task on that host is doing useful work

– System - no tasks are doing user work and at least one task is busy executing PVM system routines

– No tasks

5/30/00 CSE 225

Network

5/30/00 CSE 225

Space time

Shows status of all tasks as they execute across all hosts

• Computing - executing useful user computations• Overhead - executing PVM system routines for

communication, task control, etc. • Waiting - waiting for messages from other tasks • Message - indicates communications between

tasks

5/30/00 CSE 225

Space time

5/30/00 CSE 225

Utilization

• Summarizes the Space-Time view at each instant by showing the aggregate number of tasks computing, in overhead or waiting for a message.

• Shares same horizontal time scale as the Space-Time view

• Zooming-in • Zooming-out

5/30/00 CSE 225

Utilization

5/30/00 CSE 225

Call trace

• Displays each tasks' most recent PVM call

• Changes as program executes

• Useful for debugging

• Clicking on a task in the scrolling task list will

display that task's full name and TID

5/30/00 CSE 225

Call trace

5/30/00 CSE 225

Task output• Provides a view of output (stdout) generated by

tasks in a scrolling window • Can be saved to a file at any point

5/30/00 CSE 225

Concluding remarks

• System complexity is rising fast– computational grids– multidisciplinary applications– performance tools

• There are many open problems– adaptive optimization– performance prediction– compiler/tool integration– performance “quality of service” (QoS)

5/30/00 CSE 225

Concluding remarks

– the software problems are large & cannot be solve in isolation

– open source collaboration

– vendors, laboratories, and academics

– technology assessment