Measuring What MattersEducause 2007

Craig Blaha, Information Technology ServicesDavid Burns, McCombs School of Business

Copyright Craig Blaha and David Burns, 2007. This work is the intellectual property of the author. Permission is granted for this material to be shared for non-commercial, educational purposes, provided that this copyright

statement appears on the reproduced materials and notice is given that the copying is by permission of the author. To disseminate otherwise or to republish requires written permission from the author.

Abstract

Determining what to measure, how to measure it, and to whom to report the results can be more of an art than a science. In this session, four major research universities

will address how to communicate the importance of IT to administration, outreach, teaching and learning, and

research.• Project Overview•Lessons Learned•Moving Forward

Who We Are• Brian Busby

Collaborative Applications ManagerUniversity of Wisconsin-Madison

• Phil DevanSr. Director Network Research ServicesThe Pennsylvania State University

• Geoff LakemanDirector, Network Engineering & Technical ServicesUniversity of Washington

• Jim LoweChief Information Security OfficerUniversity of Wisconsin-Madison

• Shanna Smith, PhDDivision of Statistics and Scientific ComputationUniversity of Texas

What We Did

• Ambitious Plan– Account for all IT spending – Develop service catalogs– Identify key metrics

• Focus On Central IT Organizations• 4 Stakeholders Of Higher Education

– Administration, UW-Madison– Outreach, Penn State– Research, University of Washington– Teaching & Learning, University of Texas

Approach

• Interview key stakeholders outside of IT1. From your perspective what are the goals, strategies or

objectives you are striving to achieve to make the university a better learning and working environment (what matters)?

2. What measurements are you using to gauge the progress towards achieving these goals (how do you know you are succeeding)?

3. From your perspective, how does IT help you make your work successful (role of IT)?

4. What do you think IT should be measuring and why?

…This would be where things went off the rails.

What Happened

• Discoveries– People hadn’t had a conversation like this with

someone from IT… EVER!– Metrics don’t matter?!?

• Goals• Barriers• Role of IT

Goals

• Teaching and Learning– Reasoning , moral & ethical

thinking– Computer competency– Problem solving– Teamwork and

communication

• Outreach– Research, teaching, service– Leverage talents of

university

• Research– Research grants & contracts,

published work– Positive educational &

economic impact– Collaboration

• Administration– Efficiency, transparency,

integrity– Collaboration and

communication– Savings

Barriers

• Teaching and Learning– Many faculty married to

old learning models– Individualized, flexible

learning model is difficult with one teacher and many students

• Administration– Resistance to change– External forces– Politics and minority

stakeholders

• Research– Funding– Improving access to

resources– Available infrastructure– Highly competitive

• Outreach– Funding– Marketing/perception– Geography– Little involvement locally

Role of IT

• Teaching and Learning– Accessibility– Resource-saving– Interaction & collaboration– On-the-spot information– Student perceptions

• Administration– It’s not the technology – it’s

what we do with it– IT helps us provide services– Stakes are high - success of

IT is tied to success of administrative organization

• Outreach– Enabler and enhancer

• Videoconferencing for problem solving

• Fundamental to World Campus– Much of it should be common

good• IT is fundamental• Like a utility

• Research– Research grants depend on

technology infrastructure– Sharing IT resources from

around the world promotes big science

“Not in the least bit interested, metrics become the goal”

• Themes– Communication– Collaborative Metrics– Alignment

Communication

Communication

• “No news is good news”• Better communications from IT

– What’s going on?– Lay language– “IT has A real marketing problem”

• Listen– Understand the real mission

• Laying the groundwork– Establishing a relationship– Working towards goals, metrics, and alignment

Essential Questions• What is the value derived from IT?• What are the results that matter?

– Measuring performance against desired outcomes– Converting expectations to service standards

• How are these results measured?– Types of performance measures

• Reliability measures• Responsiveness measures• Project measures• Utilization/adoption• Client satisfaction

A Path Forward

Collaborating On Metrics

• Leadership from IT required• Starting the conversation about metrics

• Present a metric• Discuss that metric regularly

– Especially progress or problems

• See if it ends up being useful• Wash, rinse, repeat.

• It doesn’t matter unless you measure it• If it doesn’t matter – DON’T measure it!

Money Matters

Worth The

Cost?

Measures of IT changes

Measures of mission changes

Itemized IT Expenses

Current Metrics

• Reliability Measures– Quality = no errors– Uptime

• Utilization/Adoption– Blackboard usage statistics

• Client Satisfaction– Blackboard satisfaction– Annual survey

• Responsiveness Measures– Desktop support

• Project Measures– Increased funding– Technology licensing– Budgets and timelines– Statements of work

Goals and Measures

• Goals– Operational excellence– Communication– Fiscal responsibility

• Recommended Metrics– Project based metrics are fine– Qualitative better than quantitative

Barriers

• Behavior beats metrics• Metrics that don’t matter• Defensive metrics• Fear of accountability

Finding Alignment

Alignment Process

Alignment Process

• Identify Stakeholders– Planning that leaves out the middle

• Their Goals and Measures– What is important to them– How do they measure themselves

• Mapping IT Services and Metrics– What do they use– What measures do you have for those services

• Feedback and Refine– Create ongoing dialog and engagement

Key Points

• Focus on the mission• Different strokes…• Understand customer needs• Metrics are collaborative• Active listening is communication• Strategic planning – mind the gap• Sell IT!