IT Performance Measurement

8/12/2019 IT Performance Measurement

1/12

Link: http://www.rms.net/tut_perform.htm

Subject: IT Performance Management

Tutorial: Introduction To IT Performance Management And Measurement

Issue: IT organizations are being challenged to assure that their projects /activities, (1) are aligned with overall strategic goals and business objectives, and(2) deliver promised results on time and within budget. Although accustomed tomeasuring and managing technical performance (e.g., network downtime), manyIT organizations have a poor track record of measuring and managing non-technical performance (e.g., budget, project schedules, and project risk).

Purpose: This is a high-level introduction to performance management andmeasurement designed for IT professionals who have little or no experience innon-technical performance management and measurement. The major activitiesinvolved in measuring and managing the contribution of IT projects / activities toan organization's strategic goals and business objectives are discussed.

Objective: Our aim is to provide an understanding of the basic principles of non-technical performance measurement and management systems applicable to ITprojects and activities. We discuss the basic issues: (1) what is performancemanagement, and why is it of concern to IT organizations; (2) what needs to bedone to develop and implement performance management and measurement;and (3) things you should know before you undertake an IT performancemanagement initiative.

Your Comments: The goal of this web site is to provide useful information to

those who must cope with the issues; this tutorial is part of that effort. RMS wouldappreciate your comments and suggestions to help us make the information asuseful as possible. Please take a few minutes to complete the evaluation form atthe end of this tutorial.


2/12

Contents Map


3/12

Part I.

What is Performance Management?

In general, performance management refers to the use of performancemeasurementinformation to help:

define organizational goals and objectives in clear, tangible, and quantifiedterms,

develop project / activity plans designed to attain the goals and objectives, routinely monitor actual performance vs. plans, analyze significant performance deviations, advise (via routine performance reports) key managers of situations

requiring attention, formulate corrective action plans, and implement corrective actions to remedy performance deviations or modify

plans.

The goal of performance management is to assure that organizations link plansto strategic goals and business objectives, make funding decisions in light ofproject / activity benefits and outcomes that support those goals and objectives,and actively manage projects and activities to assure that the planned benefitsare realized.

In short, it is actively and proactively managing an organization to assure that itachieves pre-determined levels of performance. In contrast, management thattakes action only after a performance problem significantly affects the business isengaged in crisis management and damage control, not performancemanagement.

What is the performance management issue for IT organizations?

Many senior managers believe that IT organizations are:

wasteful and inefficient (as evidenced by chronic project cost overruns,late deliverables, etc.), and

not aligned with strategic goals and business objectives.

Senior management is often handicapped in its decision-making for IT because ithas difficulty determining:

if the most important strategic and business needs are being met if they have been presented with the best IT options for decision-making the appropriate funding levels for IT projects / activities (how much to

budget) whether the IT organization is likely to deliver the promised benefits and

results on time and within budget

Typically, these are characteristics of organizations that have paid insufficientattention to measuring and managing IT performance and results, and have


4/12

failed to establish an overall analytic framework for IT evaluation and decision-making (See the tutorial "The IT Investment Management Approach" for adescription of an extended analytic framework.)

Instituting adequate performance management is an important step that CIOsand IT organizations can take to restore their credibility. Meaningful performancemeasurement and management reporting will tell IT and senior managementwhat is working and what is not; especially important is the ability to recognizepotential problems in time to take corrective action.

Dont IT organizations already measure and manage performance?

IT organizations typically measure, monitor, and act upon data provided by abroad range of performance management tools and techniques; these effectivelymonitor and control what we shall refer to as technical performance (e.g., through"configuration management" and "fault management").

When asked about the performance management systems and processes, CIOsand IT managers will typically speak at length in terms, such as:

the polling function of the topology map provides a snapshot of overallnetwork vitality, and it can issue alerts when nodes become unavailable orare slow to respond. When a node is failing, it automatically alerts networkmanagement. To get more information on a node, we drill down for criticalstation statistics, and

the protocol distribution function tracks total packets, octets, percentutilization, packet rate, and average packet size.

Management of technical performance is an important part of an ITorganizations job; there is no doubt about that. But, the IT organizationsmanagement responsibility doesnt stop there; IT managers are responsible forassuring that they are meeting their organizations highest priorities and businessneeds to the extent possible with the available resources.

This performance information that most IT organizations lack relates to the non-technical aspects of the job; the information that can tell IT and seniormanagement whether:

1. IT projects / activities are progressing as planned

2. there are telltale signs of future performance problems3. there are troubled projects / activities that require higher managements

intervention, or4. actions already taken to correct problems actually worked.


5/12

Part II.

IT Performance Management - What are some of the key terms andconcepts?

Before we proceed further to describe how non-technical performancemanagement works, some key terms and concepts should be discussed. Someof the terms used throughout this tutorial will be interpreted differently by differentreaders; therefore, a glossaryof performance management terms is provided.These are meant to be useful interpretations, not standard definitions.

Three of the concepts underlying performance management, which often confusenovices, are an organizations "management system", performance measures,and performance metrics:

A management system is an interconnected set of processes, and aprocess is a set of activities that produce products or services (results).For performance management and measurement purposes, products andservices are treated alike. That is, the output of a process might be aproduct (such as a computer or a project deliverable) or a service (such asPC support help desk assistance). Products and services are tangible,measurable, and susceptible to analysis; thus,

Performance measures are indicators that can be systematically trackedto assess progress made in achieving predetermined objectives (e.g., thenumber of NT workstations deployed to replace Unix workstations) orservice levels (e.g., help desk call resolution time).

Performance metrics are standards of measurement (such as minimum

acceptable elapsed time to resolve help desk calls). A metric establishes abenchmark target that is compared to actual performance - the differencebetween the benchmark (the plan) and actual performance providesinsight into what is working as planned and what is not.

Dealing with these concepts can become complicated and are often a bitintimidating and confusing at first. What is important is recognition thatperformance management is a sophisticated management tool and that we areintroducing you to some of the basics.

How does non-technical IT performance management work?

Performance management involves the routine, and sometimes non routine,measurement of key aspects of IT project / activity performance and making thisinformation available to decision-makers. The goal is to assure that the benefitsof IT are realized as planned.

Preparatory steps for performance management initiatives include: definition ofthe overall analytic framework, formulation of implementation plans, assignmentof analytic responsibility, staff and management training, and conceptual designof the necessary supporting systems applications.


6/12

Once the foundation is established, performance management involves anumber of recurring steps:

1. Gather performance data on those variables of interest to IT and seniormanagement.

2. Analyze the data to determine normal (baseline) levels.

3. Determine appropriate performance thresholds for each importantvariable; exceeding a threshold indicates a problem worthy of attention(e.g., achievement of a major project milestone is 30 days, or more,behind schedule) and possibly action.

4. Periodically (often monthly) monitor performance variables for deviationsfrom established thresholds (a.k.a., variances). When a performancethreshold is exceeded (e.g., vacant PC support staff positions increases to10% when the acceptable vacancy threshold is 3%), analysts identify avariance and determine its significance. Depending on a number of factors

(e.g., severity or risk), analysts may take one of several actions, including:

identifying the project / activity as "at risk" and more closelyanalyzing performance during the next reporting period

contacting the project / functional manager for a verbal varianceexplanation

"margin-left: 25">investigate the cause of the variance andformulate corrective action recommendations

5. Report on performance to senior management (at least annually), andfunctional management (at least quarterly).

6. Initiate corrective action.

The foregoing steps are part of an ongoing process established to support aperformance planning, monitoring, and control system. The intent is to providemanagement with a stable, recurring "early warning" mechanism to detect andcorrect incipient performance problems before they actually become problems.

How does non-technical IT performance management work?(cont'd)

erformance management also employs ad hoc methods to assess and predict

performance. These include techniques such as simulations, trend analysis, andin-depth performance reviews:

Simulation (e.g., "what if" analysis) can be used to project how changes incertain activities might affect others.

E.g.: If the PC support group is deploying NT workstations to replace Unixworkstations very much ahead of schedule, how this will affect thesoftware development group that must convert Unix-based software foruse on NT. Such simulation can effectively alert management to


7/12

impending problems (e.g., the software development group cant keep upwith the accelerated NT conversion schedule; as a result, business userswould lose mission-critical functionality). This knowledge would enablemanagement to implement corrective measures (e.g., either put NTdeployment back on its original schedule or accelerate the softwaredevelopment process by increasing development staff).

Trend analysis can be used to reveal recurring difficulties that are notdetected by ongoing performance monitoring.

E.g.: Senior management has imposed a standard employee-to-consultantratio for all functional areas; it is the same for accounts payable as it is forthe IT organization. A trend analysis, possibly spanning several years,might reveal that the realities of the IT business consistently force the ITorganization to either violate the standard or shut-down important activitiesbecause sufficient numbers of technical personnel cant be hired. Theanalysis would tell management that the problems of the past are likely torecur unless the standard for IT is changed.

Performance reviews (a.k.a., performance audits) can provide IT andsenior management with objective, often comprehensive, assessments ofthe performance of a major function, activity, system, or process.Performance reviews (audits) are not financial audits, although they mayinclude financial elements (e.g., a functional performance review mightinclude evaluation of the adequacy of budget resources to accomplishbusiness objectives).

These are some of the routine, and not so routine, tools and techniques used inthe practice of performance management.

IT Performance Management - How to get started

Measuring performance isn't easy; too often, performance measurementinitiatives falter because:

those charged with planning and implementing the initiative had little or noprior experience with performance management and measurement (often,with the best of intentions, they repeat common, predictable, andsometimes fatal mistakes)

managements information needs were not identified in advance, and awell-thought-out conceptual framework and implementation plan was not

developed (in one case, after a years work, the first performance reportwas submitted to senior management two months late; the designers weretold "this is not the information we need to see")

too much performance information materialized too soon (theorganizations capacity to assimilate, interpret, and react to it wasoverwhelmed)

To help assure the success of a performance management initiative:

understand managements information needs


8/12

have a clear picture of what you are trying to achieve and how you will doit

involve people who have actually established and managed performancemanagement systems

study what other organizations are doing - learn what works and whatdoesnt

manage expectations - make sure all key participants understand andagree on what will be accomplished and when

use a limited pilot to gain experience (in other words, confine yourmistakes to a small area that you can fix quickly), and demonstrate earlyresults

Performance management initiatives can materially improve the way anorganization plans and conducts its business; careful preparation and adisciplined approach can make it happen.

Next, we will briefly discuss some of the other things you need to know.


9/12

Part III.Five Things You Should Know Before Undertaking An IT PerformanceManagement Initiative

Before undertaking a performance management improvement initiative, youshould be aware that:

1. they are often implemented in concert with other managementimprovements

2. improvements take time and sustained management commitment

3. adequate resources must be made available

4. the IT organization will need to learn new management techniques

5. some of the approaches recommended in current literature can be risky

1. Performance management improvements are often implemented inconcert with other management improvements

IT Performance management initiatives are often implemented by organizationsthat want to fundamentally shift the focus of management decision-making from apreoccupation with staffing levels and costs to a balanced focus including key"outcomes". Outcomes are results expressed in terms of the real difference thatan organizations work makes on the way business is conducted and what theorganization achieves.

To accomplish this, organizations sometimes undertake a radically differentapproach to planning and management. This can include development of:

Strategic Plans, containing (1) a mission statement, (2) a set of strategicgoals covering the organizations major business areas / programs andfunctions that are linked to the responsibility / program activity structure inthe budget, and (3) a description of how the organization intends toachieve these goals.

Annual Performance Plans that link the strategic goals and business

objectives with the day-to-day activities. These plans are developed aspart of the budget preparation process, become key factors in makingbudget decisions, and are adjusted to reflect budget decisions (e.g., if theIT organization planned and proposed to implement 5,000 NTworkstations and senior management decided to provide funding for only2,500, the plan would be reduced to 2,500).

Performance Reports, accompanying the annual budget request, to seniormanagement. This tells management what was actually accomplished


10/12

during the preceding fiscal period.

An IT Investment Management Approach is adopted to assure the "best"IT decisions are made. (See our tutorial, "The IT Investment ManagementApproach", for more information on this topic.)

2. Improvements take time and sustained management commitment

Effective organization-wide performance management initiatives take time todesign, develop, prototype, and fully implement; they also require sustainedcommitment from top management.

Consideration of the following questions will give you an idea of why time isneeded to plan and implement performance management in any organization:

How well are we measuring outcomes and results now, if at all?

If it is such a good idea, why arent we doing it already?

What performance will be measured? How? When do we need to do it?

How will we "link" strategic goals and business objectives to IT projects /activities?

How will we use performance information once developed?

How prepared is the organization to make changes based upon evaluationof its performance?

What do our managers and staff (central as well as IT) know aboutdeveloping, analyzing, and interpreting performance measures? Do theypossess the knowledge, skills, and abilities to implement and maintain aperformance management system?

Do we have the systems and technology necessary to routinely gather,process, and analyze performance measurement data? What is needed?

Answering these questions at the outset can be a sobering experience; yet, theyare indispensable to formulating a successful implementation plan. Once thequestions are answered, senior management buy-in is often secured by

developing a solid business case. Buy-in includes recognition and acceptance ofthe fact that this is a long-term undertaking and that unwavering top managementsupport is essential.


11/12

3. Adequate resources must be made available

Information is rarely cheap or free. Organizations that want to reap the fullbenefits of performance management must be willing to commit the necessaryresources. This includes:

design and maintenance of the process development of management and staff knowledge and skills dedication (eventually) of staff to perform the necessary analyses acquisition, or development, of the technology to capture and store large

amounts of current and historical data changing budgeting practices and procedures, along with budgeting

systems capabilities

The resources committed should be in proportion to the needs and benefits

4. The IT organization will need to learn new management techniques

Performance management can yield knowledge and information about theorganization not previously available. The ability to pinpoint potential, and actual,performance problems will force IT management to come to terms with issues ina new way (for example, what will the CIO do if a trend analysis reveals that his"star" software development project managers key deliverables are always sixmonths, or more, late). It will also require a more businesslike approach to ITproject and activity planning, control, and reporting.

Many technically trained professionals will need to learn, or relearn, managementskills and techniques that they never viewed as integral to their jobs.

5. Some of the approaches recommended in current literature can berisky.

As we have already noted, performance management and measurement aresophisticated management tools that can yield significant benefits to theorganization. A review of approaches discussed in some current publicationsreveals that quite a few are based more upon academic theory than solidimplementation experience.

For example: A recent article discussing how to develop performance indicatorsadvises readers that the best approach is to "empower" operational staff by

having them conduct "self-directed....brainstorming sessions" to identifyperformance indicators, measures, and metrics.

As anyone with extensive performance management system experience knows,the likely outcome of such a random approach will be long lists of things thatmight be measured (mostly inappropriate) but little that is useful in terms of whatshould be measured. That is not to say that operations management and staffshould not be involved; their participation and acceptance is essential to thesuccess of performance management initiatives. However, such efforts must becarefully structured, directed, and supported by expert guidance. Random, ill


12/12

considered efforts produce little that is useful and can quickly undermine thecredibility of the initiative.

The moral: Evaluate the source before you act upon the advice.

Tutorial Recap and Conclusion

In this tutorial, we have seen that:

Senior management often believes that IT organizations are wasteful andinefficient, based upon a track record of chronic cost overruns, latedeliverables, etc.

The purpose of Performance Management is to assure that IT projects /activities achieve their planned level of performance and deliver intendedresults on time and within budget; this is "non-technical" performancemanagement

IT organizations that maintain adequate performance managementsystems take a significant step toward restoring credibility

Performance management improvement efforts are often part of an overalleffort including other processes, e.g., budgeting

Performance management involves the routine, and non-routine,measurement of key aspects of IT project / activity performance

An important benefit of performance management is the ability to detectincipient performance difficulties and correct them before they become

problems

Performance management initiatives can materially improve the way anorganization plans and conducts its business. Success requires:

time, and a sustained management commitment

adequate resources (analyst, training, process, and system)

careful planning, and a disciplined implementation approach

If you are responsible for developing and implementing a performancemanagement initiative:

thoroughly research the literature

carefully evaluate your sources; many suggest questionable approaches

work with someone who has actually designed, implemented, andmanaged a performance management system - you can avoid years ofwasted effort.

Documents

IT Performance Measurement