1CSE1720

CSE1720 – Metrics

Week 3

Measurement and Management of Performance

CSE 1720 Lect / 2

Metrics

• Metrics are now heavily used for measurement and management of performance against objectives

• It is sometimes called ‘Management Science’ and relies heavily on mathematical techniques

CSE 1720 Lect / 3

Metrics

• As with many software packages, the user needs to have an understanding of their purpose, their validity and their relevance to a particular environment or application

CSE 1720 Lect / 4

Metrics• A metric is a measurement

• In IT, ‘metrics’ are measurements of inputs, outputs and processes

• The processes are widely varied – research, marketing, manufacturing, finance, ??

• Can Accounting be metricised ?

• Can metrics be established for a ‘system’ ?

CSE 1720 Lect / 5

Metrics• How is the process started ?

• A benchmark is established – generally this is arbitrary (or a stating point)

• Experts, consultants, experienced subject planners, users, controllers, auditors (for instance)

• This provides a basis for actual measurements and statistics to refine (or alter) the benchmark process – and of course its outputs

CSE 1720 Lect / 6

Metrics• Metrics are selected to provide

The most amount of information But using the least amount of resources

To report the results in the most effective and presentable manner

CSE 1720 Lect / 7

Metrics• One definition of a process is ‘ a series of actions or

operations leading to an end product’

• Another definition is an action or a continuous and regular action or succession of actions or operations occurring or performed in a definite manner

• Inputs and Outputs are also ‘part of the process’.

CSE 1720 Lect / 8

Metrics• Most processes exchange inputs and outputs

• There is an ‘interface’ between Inputs and Outputs

• There is an Interface between Accounting and Finance

• There are interfaces between Customers and a Company

• Between a regulatory Body and a Company (ICCC ? Or

the Dept. Of Fair Trading ….

CSE 1720 Lect / 9

Metrics• Processes are ‘united’ or enabled by ‘Workflow’

• This is a series of tasks which share or a connected by dependencies

• An workflow requires resources

• A task normally has a number of of operations which the resources perform on components and the result is a product (as are your assignments).

CSE 1720 Lect / 10

Metrics• Now for some daylight :-• Each operation• Task Input and Output• Interface• Dependency• Each resource • are capable of measurement

• And these measurements are the basic components for metrics which will provide ‘intelligence’ about organisation (or enterprise) performance

CSE 1720 Lect / 11

Metrics• Don’t get the wrong idea – it would be both difficult and

cost ineffective to measure every component of every operation

• And, is ‘every’ component of every operation ‘measurable ?’ What do you think ?

• A ‘measure’ must be quantifiable

CSE 1720 Lect / 12

Metrics• How are measures described ?

• They can be described using their ‘units’

• Distances can be in millimetres, centimetres, kilometres

• Mass can be described in milligrams, grams, kilograms

• Speed can be in mm/sec, metres/sec, kilometres/hour – and don’t forget the various speed restrictions when you

are driving – you could become part of a ‘metric’.

CSE 1720 Lect / 13

Metrics• The unit of measure defines the a measure, not the

entity being measured

• Single valued metrics are of very little use – there is insufficient ‘data’ to develop a pattern

For instance Earth to Mars flights (and possibly Mars to Earth flights) probably will not be frequent enough this decade to provide reliable metrics about anything.

CSE 1720 Lect / 14

Metrics• Measures are useful when they become part of a

segment of taken from a statistical sample population – such as genes, viruses, cells, neurons for instance

• When we use the word ‘normal’ we are indicating that there is some time series analysis and a comparison to a basis for ‘measurement’.

CSE 1720 Lect / 15

Metrics• Victoria’s water storages are not ‘normal’ compared to

storages of say 35 to 40 years ago – or are they ?

• Are statistics available ?

• Have conditions changed ? - population, uses ?

CSE 1720 Lect / 16

Metrics• You ‘automatically’ use ‘types’ – currency, date,

character, number and so on are data types (as in Database)

• Measures are also ‘typed’

• Some of these ‘types’ are Binary Statistical Average Ratio

CSE 1720 Lect / 17

Metrics• The Average age of students who enrol in Higher

Degrees would be an ‘Average’ type

• The number of vehicles involved in accidents (or incidents) related to the age of the vehicle, or the driver, or the make of vehicle would provide us with ratio of one measure (number of incidents) to another another measure (make) is another example of metric type

• Any suggestions of other ‘types’ ? ?

CSE 1720 Lect / 18

Metrics• Another type would be Productivity

• Which could be expressed asResource Usage/number of units produced as in the case of Building Investors where the metric is the Income (in $A) per square metre pf leased space and the leased space is shown as a metric of the total available leased space / actual leased space in square metres.

CSE 1720 Lect / 19

Metrics• Metrics can be either ‘simple’ or compound types

• Simple = singular and direct – they stand alone and are not associated with other metrics and measure single or simple aspects of the area of interest

• A case here would be time for an organisation to fully process a user application (rental/lease of a residence),

issue of a fine as in PERIN.

CSE 1720 Lect / 20

Metrics• The entity here would be ‘processing the application’ and

is measured individually for the attribute ‘time’

• There is no other interference or complexity from other ‘applications’

• Averaged over time gives the performance standard for the application processor, and then variances from the ‘standard’ can be calculated

CSE 1720 Lect / 21

Metrics• It is a ‘direct’ metric

It is not derived from another or other metrics

It is not evaluated relative to the performance of other like ‘application processors’

CSE 1720 Lect / 22

Metrics• Which leads to Compound Metrics• These are derived (also called indirect)• They may be composite• They may be hierarchical

• And overlap is possible – but this is not necessarily a good policy, and cases of overlapping metrics need to be detected and eliminated or combined

CSE 1720 Lect / 23

Metrics• It is a good rule to convert different/overlapping metrics

to a common unit of measure –

Incidence of occurrences over time Frequency Time groupings Money – costs or income (profits ?) Typical units of measure are $s/hour, $ per unit

produced, units per $ (or per $1,000 dollars) Units/hour is another unit of measure

CSE 1720 Lect / 24

Metrics

• Compound metrics are complex

• Clear understanding and careful construction are essential to the validity and integrity of the metric

• They need to be carefully built – and they must be clear

and relevant

CSE 1720 Lect / 25

Metrics• Compound Metrics can be

Weighted and Composite Averages Statistical Analyses Layered Metrics Thresholds and Triggers

Any others that you know of ?

CSE 1720 Lect / 26

Metrics Weighted and Composite Averages

• But weighted averages may not be accurately representative

• They may be skewed

• They may produce totally invalid results

• If the ‘weights’ are incorrectly based, then it is unlikely that the results will be correct

CSE 1720 Lect / 27

Metrics• A Composite Average is the mean of a group of

averages• They are generally assigned weights

• If these is applied, the weight of the composite average can be fed back to the components which make up the components of the composite average as a percentage of the total of the component average values

• This can be further investigated for sensitivity

CSE 1720 Lect / 28

Metrics• Statistical Analysis

Regression analyses Forecasting Correlation analyses Variance Analyses Co-Variance analyses

• Knowledge Workers need to be familiar with these techniques

CSE 1720 Lect / 29

Metrics• Layered Metrics

Consolidation Metrics

Each metric component has a hierarchical relationship with other members of the set

In an Analytical Hierarchical Process, Subordinate (lower level) metrics can influence Superordinate (higher level) metrics

CSE 1720 Lect / 30

Metrics• Thresholds and Triggers

This allows for action to be taken on a ‘out of range’ values or values. It can also initiate action on a value or values within a given boundary set.

Manufacturing is often associated with Statistical Process Control metrics.

High volume, long running processes result in large number of identical units (motor vehicles, parts, milk,

medicine …..)

CSE 1720 Lect / 31

Metrics• Statistical data can be either

Periodical ( every 15, 30, 60 minutes)

Quantitative ( each given number of units processed 100, 2000 ……)

Details are plotted or charted• Upper Control Limits• Lower Control Limits• Target value

• Evaluation of performance ( + or -) then triggers ‘corrective action’

CSE 1720 Lect / 32

Metrics• Alternatively, ‘trends’ (a number of increasingly

negative, or increasingly positive, measurements could

trigger a corrective action

CSE 1720 Lect / 33

Metrics• You should now notice a subtle change in emphasis• • We will now be discussing

• The relevance of a particular metric• The validity aspect• Which metrics suit which ‘businesses’

let’s move on

CSE 1720 Lect / 34

Metrics• Any discussion on relevance and validity must be based

on what effect the metric will have on the ‘business’

• You have heard of the phrase ‘at the end of the day ..’ and ‘the bottom line is ……’.

• Metrics must be chosen so that their effect ‘at the end of the day’ are both valid and relevant

CSE 1720 Lect / 35

Metrics• Business goals are a statement of the business

objectives

• The Business goals of Monash University are : Excellence in education Excellence in research and scholarship Excellence in management Innovation and Creativity

From a presentationby Vice Chancellor R LarkinsApril 2004

CSE 1720 Lect / 36

Metrics Diversity International focus Fairness Engagement Integrity Self reliance Education

CSE 1720 Lect / 37

Metrics• And there must be a method of measurement so that

performance can be evaluated

• A metric may stand alone - or it may be a member of a system of metrics - you saw these mentioned in ‘compound metrics’ which report across function or processes

• And every metric system and part or component, must support and be associated with measurable, clear business objectives or goals.

CSE 1720 Lect / 38

Metrics• If a metric supports the goals of an organisation, then

it is relevant.• If not, it is ‘irrelevant’ and is useless or misleading

• What if the goals of an organisation change ?

Could they ? Could a bank for instance change its customer base and address ‘corporate’ customers only ?

CSE 1720 Lect / 39

MetricsWhen would the metrics alter ?

How would up to date metrics be devised ?

How long after the ‘change’ would ‘new’ metrics become active ?

How would these relate to the superceded metrics ?

CSE 1720 Lect / 40

Metrics• Validity of a metric can be difficult

to determine to develop to maintain to stay focussed

• If any part of a metric becomes invalid, and this in not noticed and corrected, then the effect on a business can be fatal - do you remember the Pan-Am airline of some years ago ?

• And which is now defunct ? • What happened ?

CSE 1720 Lect / 41

Metrics• Metrics in many ways are very similar to System

Development and Implementation• They need to be given

Senior/Executive Management Support Senior/Executive Management Commitment Senior/Executive Funding

They need to be seen as a valuable and reliable tool of Senior/Executive Management

and Senior/Executive Management must be able to use these tools confidently and competently

CSE 1720 Lect / 42

Metrics• Validating Metrics: some guidelines

1. The metric must relate to the goals- The relationship must be valid- It must be meaningful- Its purpose must be understood- Is it one of a family - and if it is can they be

consolidated- Is there a mechanism to ‘audit’ the purpose of

the metric so that if or when conditions alter the metric can be examined for continued use, alteration, cancellation

CSE 1720 Lect / 43

Metrics 2. The components of a metric must be correct and

applicable– A compound metric should be ‘decomposed’ into

its parts– Each part should be ‘proven’ to be a support to

the metric– Is the metric limited in some of its applications. If

it is, why is it used ?– What is the effect of ‘weights’ as in weighted

average. Is there a procedure to assess the credibility of the results ?

– Are adjustments necessary ?

CSE 1720 Lect / 44

Metrics 3. Data Quality. This must be both ‘high quality’ and

of course ‘applicable’ data Data needs to be quality controlled - evaluated. The metrics which produce data must also be

carefully analysed - on an ongoing basis 4. Is the result of a metric applicable to its

contribution of a result ?

Or to components to which the metric is a contributor ?

A metric may contribute to an upward ‘result’ as well as being ‘supported’ by a lower level metric.

CSE 1720 Lect / 45

Metrics 5. Sensitivity Analyses These are methods of measuring or calculating the

effect of input data to a metric - does it have a linear or exponential effect ? Can ‘weights’ of components be affected by incoming data - when or if this happens is there a ‘trigger and procedure’ to handle the situation

(just as a comment, Excel’s Solver function produces a sensitivity report which is very useful as a reliability support to the ‘Solver solution’.)

CSE 1720 Lect / 46

Metrics• Quality is made up of a number of processes and

steps • The ‘steps’ to Quality are often repetitive and circular• Quality applies to a total process

• And of course the question is ‘ why does validation seem to be so important and necessary ?’

CSE 1720 Lect / 47

Metrics• You have probably noticed that ‘change’ is the

dominant factor in Business.

• Philosophically, Business must adapt to survive. Change has become a ‘constant’

Valid today - Tomorrow invalid or irrelevant

CSE 1720 Lect / 48

Metrics• Indicators of performance must reflect the ‘current

state’• This impacts on continuing relevance and validity of

performance metrics• This requires continuing audit and maintenance• And in turn, any action taken as a result of a metric will

be ‘probabilistically sound’.

So now let’s have a deeper look into Organisational Metrics

CSE 1720 Lect / 49

MetricsResponsibility Accounting which occurred in the 1970’s created the need for Managers to have accurate and representative monitoring tools.

Executives also must have a similar but higher level tools or tools (have you heard of ‘dashpots’ ?)

Bottom up design of Organisational metrics (ascending levels of management) will succeed if there is a clear expression from Senior Management of those metrics which are seem to be essential to support the ‘Business Plan’

CSE 1720 Lect / 50

MetricsAt this stage you could refer to the Monash Vice-Chancellor’s plan for this University

In Business, it is generally recognised that there are 4 levels or layers in an organisation (another term is Enterprise)

In Business, these layers are ‘Executive, Division, Department, Group’

In Monash they are Executive, Faculty, Department and School

CSE 1720 Lect / 51

MetricsEach level must have policies and measurable benchmarks to ensure that no component is ‘out of step’

Benchmark standards: Internal and External

Internal benchmarks are the formal targets which must be met by business processesThey measure progress to meet an objective by :-

Clear definition of the objectiveIdentification of appropriate and applicable metricsMeasuring and monitoring processes and

progress

CSE 1720 Lect / 52

MetricsExternal metrics have a slightly different - they compare an Organisation (or Company’s) internal metrics with or against those of other Companies’

This gives a measure of performance against Industry standards.

CSE 1720 Lect / 53

MetricsMetrics are systems - and like systems they have a

‘metrics life cycle’

This life cycle defines the procedures and controls about

- Analysis

- Design

- Development

- Application processes - monitoring

- Maintenance - changes/modifications as ‘externals’ change

- Close Down, and Replacement

CSE 1720 Lect / 54

MetricsHow is a Metric selected ?There are a number of ‘qualifiers’ as with any System

1. Cost / Benefit ratioCosts are incurred at the initial strategy

development, and equally or perhaps more importantly during the whole of the metric life cycle.

These costs need to be offset by time and cost savings, and in improvements in quality and efficiencies.Benchmarks are used to evaluate or calculate the return on investment - which quantifies the derived benefits.

CSE 1720 Lect / 55

MetricsMetric TolerancesAny form of ‘accurate’ data is subject to fault or risk tolerance acceptance.In some of the material covered, it was mentioned that benchmark standards should be established before a

metrics program was started the benefit of a metric needs to be quantified (return on

investment for instance) If standards are not developed, then ‘estimates’ may

need to be considered Estimates may not be acceptable If management has a high risk tolerance, the more likely

that estimates will be acceptable - is this good Policy ?

CSE 1720 Lect / 56

MetricsTolerances can

• mask or hide a problem• cause problems at a high and probably ‘out of

control’ rate when (or if) set boundaries are exceeded - or if the ‘boundaries’ are continually altered

Metrics are ‘agents’ which which either highlight or discourage some ‘adverse’ behaviour

However - this may be dangerously simplistic - an individual metric can be seen as ‘rewarding’ or ‘discouraging’ behaviour’

CSE 1720 Lect / 57

MetricsA more enlightened approach :

Use complementary (two or more) metrics to a specified process - this will highlight any inconsistencies or difficulties

On an Impact basis - Processes are invariably linked and an change in one process will most likely result in a ‘bounced’ impact on another

• If less of item A is produced, then there could be less consumers, and less revenue …...

CSE 1720 Lect / 58

MetricsAs you have gathered, Metrics (or their installation and use)

can be costly ‘overheads’It is not surprising that many Companies keep a database (or

a repository) of the type or types of metrics used the conditions which each metric addressed the application area - the divisions, departments

mentioned earlier a description of the functions of the metric (also known

as functionality) a map of the inputs, streams, interactions, outputs of

connected inputs Specialist or derived formulae - and what the formula

basis was

CSE 1720 Lect / 59

Metrics• A few words about output

This identifies the success (or failure) of the use of the metric (or more likely, metrics)

And of the value of the particular metric to the next lower level metric

Why so this ? - Metrics are rarely run continually - they can be thought of as an audit - and repeated at irregular(?) intervals. So it is important to NOT re-invent the wheel.

CSE 1720 Lect / 60

MetricsChange is the keynote of our business environment

Information Technology seems to provide (at a cost) opportunities in collection , processing and distribution of data and information

And what about ‘Senior Management ? The Executives They find themselves caught up in the race to achieve

‘faster’, cheaper’ and ‘better’ - as you no doubt will.

Executive Performance is demanded by Shareholders, as in the recent case of Enron and the National Australia Bank

CSE 1720 Lect / 61

MetricsYou have probably heard of, or read about, these :BPM - Business Performance ManagementBAM - Business Activity ManagementKPI - Key Performance IndicatorsOLAP - On Line Analytical Processing

all of which are relatively new tools - but what will be their replacement in the next 3 to 5 years ?

You have seen the emergence of the title CIO - Chief Information Officer - sometimes and Executive level position - what is their function ?

CSE 1720 Lect / 62

MetricsBasically their function is to make sure that the Company, Organisation or Enterprise for whom they work is able to obtain the benefits of rapidly changing technologies.

Is it likely that if the CIO is seen to be ‘out of touch’ that Executives will deal directly with 3rd party vendors ?

Is there a risk here ? Can it(they) be quantified ? By whom ?

Management is largely based on the ability to ‘measure’ - and metrics provide this capability

CSE 1720 Lect / 63

MetricsMetrics, by their nature, call for or cause a much closer linkage or affinity between Information Technology and Business

Business objectives and strategies support (if not demand) that Information Technology understands these criteria

How is this achieved ? Some suggestions:1. Common definition of terms - not just words2. Develop a metrics ‘information base’ 3. Model the Business and Business processes as of now - not how it was 3 to 5 years ago - and keep remodelling to ensure accuracy

CSE 1720 Lect / 64

MetricsIdentify areas as previously mentioned where metrics

offer opportunities - don’t guess, model and observe the advantages and disadvantages

Observe and heed the cost/benefits

Carefully assess the areas of application of metrics

Set up a ‘pilot’ scheme - and let people know what ishappening and why - get them onside

CSE 1720 Lect / 65

MetricsSet up the means for assessing the benefits/costs

Review the processes, their effects, their effectiveness

Refine or replace the metric method(s) if necessary

and (again) Communicate. Let others in the Organiation know about the effectiveness of the use of metrics used - and also of lack of effectiveness if this is the case