Emerging Trends in Institutional Assessment: What Works and What Doesn’t

June 14, 2004

Chet WarzynskiCornell University

CAUBO 2004

Objectives

Describe three different approaches to institutional assessment

Examine the implications of culture in relation to institutional assessment

Present some best practices for implementing an assessment system

Drivers of Assessment New mandates for knowledge to address societal issues

Increasing costs of higher education Public demands for accountability and comparison Need for continuous improvement and change Stakeholder dissatisfaction, e.g., students and employers Information for management decisions Need to meet accreditation requirements Motivation of people

Developing An Assessment System

Define Core Values and Goals

Develop Strategies and Action PlansDevelop Strategies and Action Plans

Implement Management by Assessment

Regularly test

measures against

strategies

Identify Outcomes and Deliverables

Align Work Processes, Jobs, and Behaviors

Design Assessment System with leading, lagging, cost control, & value creation measures

Systems Model of the University

Input Throughput Output• People• Information• Finances• Facilities

• Culture• Roles/Structure• Work Processes• Technology• Informal Network

• Programs & services• Revenues• Reputation/Image• Patents/Licenses

Resources Structure/Process Results

Feedback / Control

Leadership

Decision making Problem-solving

Environment

Assessment MeasuresInput Measures

Standardized test scores & school rank Number of National Merit Scholars Students-to-faculty ratio Undergraduate class size Student credit hours taught by faculty Graduate enrollment Faculty salaries by discipline and rank Graduate student stipend level Appropriations, tuition, fee revenues per

FTE student Student financial aid per student Sponsored funding per faculty FTE faculty Private giving and endowment value

Output Measures Student retention and graduation rates Graduates' career placement Advanced study enrollment Number of doctoral degrees per year Student indebtedness upon graduation General expenditures per FTE student Student credit hours per FTE faculty Demography of underrepresented population R&D expenditures as a share of operating

expenditures National Academy/prestigious memberships National ranking of academic programs Number of license agreements & patents Participation rate of alumni as donors

Cornell’s Interrelated Missions

Research

- Discovery of knowledge

Teaching

- Distribution of knowledge

Public Service

- Application of knowledge

Strategic Goals

GOAL 1 — RESEARCH: Achieve and sustain preeminence in research

GOAL 2 — TEACHING: Attain and preserve excellence in teaching

and learning through programs of superior quality and value. GOAL 3 — PUBLIC SERVICE: Effectively address the needs of

society through public service

GOAL 4 – PERFORMANCE: Increase efficiency and effectiveness

Strategic PlanCore Values

LeadershipServiceScholarshipCreativityFreedomIntegrityExcellenceCollegialityStewardshipInnovation

Mission

Cornell is a learning community that seeks to serve

society by educating the leaders of tomorrow and extending the

frontiers of knowledge.

Vision

We pursue understanding beyond

the limitations of existing knowledge,

ideology, and disciplinary structure,

and cultivate the enrichment of the

human mind and spirit.

Strategies and MetricsResearch

Create incentives that encourage faculty productivity in research and scholarship

Metrics

• Research and Development (R&D) expenditures as ashare of educational and general (E&G) expenditures• Number of publications/citations in refereed journals• Sponsored funds per FTE faculty

TeachingIncrease the quality of faculty and enhance their participation in education and learning

Metrics

• Top faculty from peer institutions• Number of courses and credit hours taught by faculty• Number of formalized instructional improvement programs and participants

Public ServiceIncrease partnerships to enhance public service, commercialization, and

entrepreneurial initiatives.

Metrics

• Number of license agreements and patents fortechnology transfer• Number of start-up companies• Number of regional technology centers• Number of partnerships

Strengths & Weaknesses

Easily Understood Flexible Computing ease Accuracy

Low tolerance for complexity

Lag indicators High transaction

costs

What Is The Balanced Scorecard?

The balanced scorecard is a 4-dimensional management system to help a business focus on achieving financial results and on creating future value through strategic activities

It translates mission and strategy into four dimensions--customer, financial, internal processes, and innovation and learning--and seeks measures for them

The Four Measurement Categories

FINANCIAL: How do we look to shareholders?

INTERNAL PROCESSES: What processes must we excel at to achieve our objectives?

INNOVATION AND LEARNING: How can we continue to improve and create value with employees, customers, and processes?

CUSTOMER: How do we become our customers’ most How do we become our customers’ most valued supplier?valued supplier?

EXAMPLE OF UNIVERSITY EXAMPLE OF UNIVERSITY STRATEGY*STRATEGY*The Revenue Growth Strategy

“Improve stability by broadening the sources of revenue from current sponsors.”

The Productivity Strategy“Improve operating efficiency by shifting to more cost-effective self-service channels for students.”

Broaden Revenue

Mix

Improve Operating Efficiency

Improve

Returns

Fin

an

cia

l P

ers

pect

ive

Develop Strategic

Skills

Access to Strategic

Information

Align Personal

Goals

Increase Faculty/Staff Productivity

Learn

ing

&

Gro

wth

P

ers

pecti

ve

Increase Sponsor Confidence in our ability to teach, research & serve

Increase Student Satisfaction Through Superior Execution

Sta

keh

ol

der

Pers

pect

ive

Understand

Stakeholder

Needs

ImproveExisting services

Develop New

Products

Shift to Self

Service Channels

Minimize Problems

Provide Rapid

Response

Inte

rnal

Pers

pect

ive

*Adapted from Kaplan & Norton, 1996

UNIVERSITY BALANCED SCORECARD*UNIVERSITY BALANCED SCORECARD*

Strategic Objectives

Strategic Measurements(Lag Indicators) (Lead Indicators)

FinancialF1 - Improve returnsF2 - Broaden Revenue MixF3 - Reduce Cost Structure

Return-on-EndowmentRevenue GrowthOperating Expense Change

Revenue Mix

StakeholderC1 - Increase student satisfaction with our curriculum & facultyC2 - Increase satisfaction on completion of studies

Share of Segment

Student Retention

Depth of Relationship

Satisfaction SurveyInternal

I1 - Understand our faculty/staffI2 -Create innovative servicesI3 – Develop new productsI4 - Shift students to cost-effective self-service channelsI5 - Minimize operational problemsI6 - Responsive service

New Sources of RevenueCross-Sell RatioChannel Mix Change

Service Complaint RateReduce cycle time

New Product and Self-Service DevelopmentHours of Service

LearningL1 - Develop core skillsL2 - Provide strategic informationL3 - Align personal goals

Employee SatisfactionRevenue per FTE

Strategic Job Coverage RatioStrategic InformationAvailability RatioPersonal Growth Alignment (%)*Adapted from Kaplan & Norton, 1996

CORNELL’S BALANCED SCORECARD

Stakeholder Satisfaction

Are stakeholders highly satisfied with administrative products & services?

Policy Development & Application

Do our administrative policies provide value to clients for services they are willing to pay for?

Employee Productivity & Competence

Are employees investing their time in work important to their stakeholders? Do employees have proper training to do their job efficiently?

Operational Efficiency

Is our organization structured & operating in the most efficient manner possible?

Best-ManagedUniversity

SELECTED SCORECARD METRICS

Stakeholder Satisfaction

Metric: Indices of customer satisfaction, timeliness, accuracy, & effectiveness

Policy Development & Application

Metric: Percent of a list of policies or parts of policies that generate non-value-added procedures

Employee Productivity & Competence

Metric: Percent of administrative workforce that demonstrates a high level of competence with systems

Operational Efficiency

Metric: Cost of administrative FTEs

Best-ManagedUniversity

Balanced Scorecard Dash Board Indicators*

*From Ruben, Brent (2004). Pursuing Excellence in Higher Education. New York: Jossey-Bass, p. 105.

IMPLEMENTATION STRATEGY*IMPLEMENTATION STRATEGY*

1. Build an Executive Leadership Team to Mobilize Change

2. Translate the Strategy into Operational Terms

3. Link and Align the Organization Around Its Strategy

4. Make Strategy Everyone’s Job

5. Link Strategy and Budgeting

6. Make Strategy a Continuous Process*Kaplan & Norton, 1999

Strengths & Weaknesses

Operationalizes strategy Focuses the entire organization Integrates diverse programs Breaks higher level measures

down into lower levels Provides a comprehensive and

systems view

Lacks environmental and competitive analysis

Lacks inclusive measures Linkage to annual resource

allocations/budgets Retention of commitment Lacks long-term view

From “Education Criteria for Performance Excellence,” Baldrige National Quality Program, www.baldrige.nist.gov

Canadian Award for Excellence*

*From National Quality Improvement website: nqi.com

Baldrige & NQI Core Values & PrinciplesBaldrige Core Values & Concepts

Visionary leadership Learning-centered education Organizational and personal learning Valuing faculty, staff, and partners Agility Focus on the future Managing for innovation Management by fact Social responsibility Focus on results and creating value Systems perspective

NQI Principles Leadership through involvement Primary focus on clients/stakholders Cooperation, teamwork, and

partnering Process-oriented and prevention-

based strategy Factual approach to decision making Contribution of each and every

individual Continuous improvement of methods

and outcomes Obligations to stakeholders, including

social responsibility

Baldrige 2004 Categories(Items Point Values = 1000)

1 Leadership 1201.1 Organizational Leadership 701.2 Social Responsibility 502 Strategic Planning 852.1 Strategy Development 402.2 Strategy Deployment 453 Student, Stakeholder, and Market Focus 853.1 Student, Stakeholder, and Market

Knowledge 403.2 Student and Stakeholder Relationships and

Satisfaction 454 Measurement, Analysis, and Knowledge

Management 904.1 Measurement and Analysis of Organizational

Performance 454.2 Information and Knowledge Management 45

5 Faculty and Staff Focus 855.1 Work Systems 355.2 Faculty and Staff Learning and Motivation 255.3 Faculty and Staff Well-Being and Satisfaction 256 Process Management 856.1 Learning-Centered Processes 506.2 Support Processes 357 Organizational Performance Results 4507.1 Student Learning Results 1507.2 Student- and Stakeholder-Focused Results 607.3 Budgetary, Financial, and Market Results 607.4 Faculty and Staff Results 607.5 Organizational Effectiveness Results 607.6 Governance and Social Responsibility Results 60

U of W – Stout’s Strategic Goals*

1. Offer high quality, challenging academic programs that influence and respond to a changing society.

2. Preserve and enhance our educational processes through the application of active learning principles.

3. Promote excellence in teaching, research, scholarship,and service.

4. Recruit and retain a diverse university population.

5. Foster a collegial, trusting, and tolerant environment.

6. Provide safe, accessible, effective, efficient, and inviting physical facilities.

7. Provide responsive, efficient, and cost-effective (educational support) programs and services.

*See www.uwstout.edu

U of W – Stout’s Objectives & Measures

NQI ROAD MAP TO EXCELLENCE*

1. Appoint Program Champion (Senior Leadership Overview)

2. Conduct base line study3. Apply NQI Framework for excellence4. Begin NQI 4-level progressive excellence

program5. Focus on continuous improvement6. Re-certify PEP level 4 every two years

*From National Quality Improvement website: nqi.com

Strengths & Weaknesses

Accelerate improvement efforts

Gain outside perspective

Learn from feedback process

Focus on results

Internal view High learning curve Low tolerance for noise

in data High transaction costs

Typology of Organizational Cultures*

Type Sociability Solidarity Characteristics

Networked High

Low

Strong sense of individualism; connections through relationships and alliances; high reciprocity; low organization commitment. highly political

Mercenary Low High Strong identification with org.

and commitment to goals; work-related social relations; top-down management; in-tolerant of poor performance.

Fragmented Low Low Low interdependence & org. identity; lack of relationships; high levels of dissent; low affect and commitment to the organization.

Communal High High Strong organizational identity; high collaboration and team-work; similar beliefs & values; sharing of work, risks and rewards.

*Rob Goffe & Gareth Jones, What holds modern company together? Harvard Business Review, Nov-Dec'96.

Business vs Academic Cultures*(Competing Subcultures in Higher Education)

• Narrow Business Mission• Market Driven• Action/Results- Oriented

(Short term)• Planful/Methodical• Hierarchical• Change by Decision• Low Tolerance for Ambiguity• Authority Based• High Accountability

* © C. Warzynski, Cornell University

• Broad Societal Mission• Faculty Driven• Research/Teaching- Oriented

(Long term)• Open/Emergent• Free Intraprise• Incremental Adaptation• High Tolerance for Ambiguity• Consensus Based• Low Accountability

New Institutional Economics* Interaction Thesis:

“If formal rules (of the new model) are in harmony with informal rules (of the culture), the incentives both create will tend to reinforce each other. A harmonious interaction of formal and informal rules reduces the transaction costs of maintaining and protecting the rules of the game, and frees resources for the production of wealth.

When formal rules conflict with informal rules, however, their respective incentives will tend to raise the transaction costs of maintaining and enforcing the prevailing institutional environment, and therefore reduce the production of wealth in the community.”

*S. Pejovich, “Toward a Theory of the Effects of Interaction of Formal and Informal Institutions on Social Stability and Economic Development,” Third International Conference Enterprise in Transition, University of Split, Faculty of Economics, Split Croatia, May 27, 1999; D. Lal, Unintended Consequences: The Impact of Factor Endowments, Culture, and Politics on Long-Run Economic Performance. Cambridge, MA: MIT Press, 1998; D. North, Institutions, Institutional Change and Economic Performance. Cambridge, MA: Cambridge University Press, 1990.

Institutional Economic Performance

Decreased Resources(Less Wealth)

AcademicModel

BusinessModel

Interaction

Reinforcing rules = low transaction costs

Conflicting rules = high transaction costs

Increased Resources (Greater Wealth)

Institution/Assessment System Fit†

† Adapted from V.Dhar and R. Stein, Intelligent Decision Support Methods, Saddle River, NJ: Prentice Hall, 1997, p. 21

Organization Culture

Model Related

Accuracy

Explainability

Response Speed

ScalabilityCompactness

Flexibility

Ease of UseEmbeddability

Independence from Experts

Computing Ease

DevelopmentSpeed

LearningCurve

Tolerance forNoise in Data

Tolerance forComplexity

Tolerance forSparse Data

Q

uality

Rel

ated

Cons

train

t Rel

ated

Transaction costs

Evaluation of Assessment SystemA. Quality-Related Questions

1. Does the system provide optimal solutions in terms of accuracy and “goodness of fit with the culture?”

2. Do decision makers need to know how the answers are derived?

3. Does the system provide answers and decisions in a reasonable amount of time?

B. Model-Related Questions

1. How flexible is the system in allowing specification changes?

2. How scalable is the system?3. How easily can the system be embedded

into a larger system?4. How compact is the system?5. How easy is the system to use?

C. Organization-Related Questions

1. Is there good, high-quality electronic information available?

2. Is there a lot of electronic data available?3. Is the organization far enough up the learning

curve?4. How subtle and easily understood are the

interactions between the variables?D. Constraint-Related Questions

1. What is the need for and access to experts?2. Are the computing infrastructure resources

adequate?3. What development time can the organization

afford?4. What are the transaction costs? Is it worth it?

Comparing Approaches to Institutional Assessment

Approach Strengths Limitations Culture Fit Strategic Planning

Explainable

Flexible

Computing ease

Accuracy

Ease of use

Low tolerance for complexity

Lag indicators

High transaction costs

Mercenary/

Fragmented

Balanced Scorecard

Precise focus

Lead & lag indices

Response speed

Embeddable

Scalable

Compactness

Ease of use

I nternal view

High learning curve

Low tolerance for noise in data

High transaction

costs

Networked/

Mercenary

Baldrige/

NQI Assessment

Broad criteria & scope

External assessment

Comparative data

Flexible

Normative

High complexity

High transaction costs

Lacks ease of use

Communal/

Networked

A Good Assessment System . . .

Clearly connected to organizational goals and strategies Focuses on strategy and establish priorities Links long term with short term Combines lead and lag indicators Links and coordinates functions horizontally & vertically Drives change, motivates action Contains effective metrics with assigned accountabilities Links to databases and information systems Can be easily understood & communicated

Best Practices in Institutional Assessment Is used as a “problem detector” Points to action that needs to be taken Is carefully designed & reviewed regularly Informs resource allocation decisions Is publicly visible across the organization Is part of a change context Is supported by the culture Guides the way managers’ manage and leaders’ lead

Lessons learned Assessment does not ensure organizational success (too many

intervening variables) Establish clear goals and strategies for the organization before

embarking on assessment Consider transaction costs of assessment (It requires

considerable time and “socialization” of the organization’s members, and stable leadership

Provide in-depth training and support to managers Align assessment to your performance management system (an

assessment system is irrelevant if there is no accountability) Evaluate and update your assessment system regularly

Management “Do’s” Make each goal, objective, and strategy clear and ensure

everyone has the same understanding of it Make sure strategies are right for the organization Choose measures at the proper level Make sure measures are “need-to-haves”, not “nice-to-

haves” Make sure metric owners are committed to their roles Do something, even if only to establish a baseline