1 “University Values – University Integrity” Integrity in Higher Education: Quality Improvement versus Ranking Improvement Nachum Finger Ben-Gurion University

1

“University Values – University Integrity”

Integrity in Higher Education: Quality Improvement versus Ranking Improvement

Nachum FingerBen-Gurion University of the Negev and the Council for Higher Education, Israel

21st International Conference on Higher Education (ICHE)

August 26-28, 2010

Trier, Germany

2

Table of Contents

The Role of the University Habermas’ essay Implications to quality

Rankings of HEI Rankings examples Methodology issues Integrity issues

The Israel HE System Quality Evaluation Pressure to Rank

Summary

The Role of the University…

In his essay “The University in a Democracy – Democratization of the University” Habermas refers to the following news item as it appeared in the Frankfurter Allgemeine Zeitung of January 11, 1967 to discuss the role of the University…

From: Toward a Rational Society, Jürgen Habermas, Heineman Educational Books, London 1971.

3

The News Item…

“In the vicinity of Sde Boker in the Negev, Israel's large desert, Ben-Gurion wants to found a university town to serve the exploitation of this desert area. The new town is being planned for ten thousand students and the corresponding number of faculty and is to bring Israeli youth into contact with the development of the desert through the acquisition of the necessary knowledge of the natural sciences and technology. It is intended primarily to develop the trained personnel who will be necessary for future industry in the desert. In particular, the development of such industry will involve enterprises that require much scientific knowledge and little raw material.”


4

The Role of the University…

“First, the university has the responsibility of ensuring that its graduates are equipped, no matter how indirectly, with a minimum of qualifications in the area of extrafunctional abilities…”

“Second, it belongs to the task of university to transmit, interpret, and develop the cultural tradition of the society...”

“Third, the university has always fulfilled a task that is not easy to define; today we would say that it forms the political consciousness of its students…”


5

Our perception as to the role of the university relate directly to the importance we attach to the idea of ranking and especially to Quality Improvement vs. Ranking Improvement as reflected in following examples and citations.

6

7

Methodology

SJTU Academic Ranking of World Universities

Nobel laureates (staff) 20% Nobel laureates (alumni) 10% Highly cited researchers 20% Articles published 20%

Articles cited 20% Size 10%

From: A Faustian Contract: Institutional Responses to National and International Rankings. Peter W A West OBE. IMME Conference, September 8-10, 2008, Paris

8

Methodology

THE-SQ World University Ranking

Student-staff ratio 20% Recruiter survey 10% Peer survey 40% International staff 5%

International students 5% Articles cited 20%

From: A Faustian Contract: Institutional Responses to National and International Rankings. Peter W A West OBE. IMME Conference, September 8-10, 2008, Paris

9

MethodologyU.S. News and World Report College Rankings

Peer assessment 25% Percentage of classes with fewer than 20 students 6% Percentage of classes with more than 50 students 2% Average faculty salary 7% Percentage of professors with highest degree in field 3% Student / faculty ratio 1% Percentage of professors who are full-time 1% Spending per student 10% Percentage of students in top 10% of high school class 6% Student SAT scores 7.5% Acceptance rate 1.5% Graduation rate 16% Retention rate 4% Alumni giving rate 5% Graduation rate performance (predicted versus actual) 5%

From: America’s Best Colleges: 2007 Edition, U.S. News and World Report LP, 2006.

10

Some Questions about Ranking Methodologies

1. Choice of indicators2. Arbitrariness of weights3. Formula changes4. Reliance on polls –

a) Who is polledb) What weight

5. Statistical validity6. Use of Quality Assessment as input7. Inconsistencies between ranking methodologies

Source: Ranking in Higher Ed. Institutions. Anthony Stella & David Woodhouse. AUQA,

2006.

11

Questions continued…

1. Bias against Humanities and Social Sciences

2. Inconsistent classification of institutions

3. Inappropriate measures of teaching quality.

Source: Richard Holmes, Asian Journal of University Education 1(1), 2006, 1-14.

12

“In 2004 the oldest public university in Malaysia, the University of Malaya, was ranked by the Times Higher Education Supplement at No. 89 in the world. The vice-chancellor ordered huge banners declaring “UM a world’s top 100 university” placed around the city. But last year the THES changed the definition of Chinese and Indian students at UM from international to national and the university’s position in the reputational surveys that comprise 50 percent of the THES index also declined. The result was that UM dropped from 89 to 169. The university’s reputation abroad and at home was in free fall. When the VC’s position came up for renewal by the Government last March, he was replaced.”

Source: Simon Marginson, “Rankings Ripe for Misleading. ”The Australian. December 6, 2006.

13

“Many of our American colleagues say that they would like to resist the rankings, but fear it can’t be done, especially if only a few institutions act. A growing number of Canadian institutions began to raise the same alarm, ultimately resulting in 25 of our 90+ institutions – including many of our leading universities – banding together to take just such a stand against the fall rankings issue of Maclean’s, our Canadian equivalent.”

Source: Indira Samarasekera, “Rising Up Against Rankings.” Inside Higher Ed. April 2, 2007.

14

“However, global comparisons are possible only in relation to one model of institution, that of the comprehensive research intensive university, and for the most part are tailored to science-strong and English-speaking universities. Neither the Shanghai nor the Times rankings provide guidance on the quality of teaching.”

Source: Simon Marginson, Marijk van der Wende. (2007) “To Rank or To Be Ranked: The Impact of Global Rankings in Higher Education”. Journal of Studies in Intl Ed.

15

“Performance indicators currently used by higher education institutions are generally chosen because they are readily quantifiable and available, and not because they accurately assess the quality of teaching (Burmons, Brouwer, Veld and Marthens, 1987). Therefore over-interpreting performance indicators is even more dangerous (Chalmers, 2007).

Source: Fabrice Henard, “Learning our Lesson” Review of Quality Teaching in Higher Education, OECD, 2010, p. 81

16

On Feb. 17, 2010 ABA President Carolyn Lam commissioned an examination of the ranking of law schools. Of the adverse effects of US News ranking – three of greatest concern:

The current methodology tends to increase the costs of legal education for students

The current methodology tends to discourage the award of financial aid based upon need

The current methodology tends to reduce incentives to enhance the diversity of the legal profession.

Source: Report of the Special Committee on the U.S. News and World Report Ranking– Section on Legal Education and Admission to the BAR

17

Just recently… “Like other observers of global university rankings, I’ve

been intrigued by the trash talk in recent months between the British publication Times Higher Education and the higher ed research-consulting group QS (short for Quacquarelli Symonds). For six years, beginning in 2004, the two organizations worked together to produce the influential and oft-condemned World University Rankings, the chief rival to the almost-as-controversial Academic Rankings of World Universities inaugurated the previous year by Shanghai Jiao Tong University. But last October, Times Higher abruptly announced that it was parting ways with QS and would instead completely revamp its ranking in partnership with Thomson Reuters, the global information firm.”

Source: Wildavsky, B. “Global Rankings Smackdown!” The Chronicle of Higher Education. July 15, 2010.

18

Contribution of Rankings “…While HE leaders are concerned about the

impact of rankings, they are also increasingly responsive and reactive to them. In addition, key stakeholders use rankings to influence their decisions: students use rankings to ‘shortlist’ university choice, and others make decisions about funding, sponsorship and employee recruitment. Rankings are also used as a ‘policy instrument’ to underpin and quicken the pace of HE reform.”

Source: Hazelkorn, Ellen. (2008). “Learning the Live with League Tables and Ranking: The Experience of Institutional Leaders.” Higher Education Policy. 21, 193.

19

Contribution continued…

1. Majority of students use rankings in their decision-making.

2. Employers look at ranking.3. Leaders in HE take action based on ranking.4. Governments/Sponsors sometimes make

strategic decisions based on rankings.

Source: Ellen Hazelkorn. “What Have We Learned About and From Rankings.” CHEA Annual conference, Jan 2010.

20


“Rankings are significant drivers of a school’s reputation. Good performance can double inquiries and applications and allow schools to charge prestige premiums. Financial Times top decile MBA programmes charge, on average, just below $80,000 for an MBA. Bottom decile schools charge only $37,000…

This paper finds that it is impossible to challenge the criteria set out by a variety of rankings organisations and it is ill-advised to boycott rankings. Schools are advised to consider which criteria reflect areas needing improvement and to continue “playing the game.”

Source: Peters, Kai. (2007). “Business School Rankings: Content and Context.” Journal of Management Development.

21


“We found that moving onto the front page of the U.S. News rankings provides a substantial boost in the following year’s admissions indicators for all institutions. In addition, the effect of moving up or down within the top tier has a strong impact on institutions ranked in the top 25, especially among national universities. In contrast, the admissions outcomes of liberal arts colleges – particularly those in the lower half of the top tier – were strongly influenced by institutional prices.”

Source: Bowman, N., Bastedo, N. (2009). “Getting on the Front Page: Organizational Reputation, Status Signals, and the Impact of U.S. News and World Report on Student Decisions.” Res High Ed.

22

Integrity: Playing the Rankings Game “For ten years Reed has declined to fill out the annual

peer evaluations and statistical surveys that U.S. News uses to compile its rankings. It has three primary reasons for doing so. First, one-size-fits-all rankings schemes undermine the institutional diversity that characterizes American higher education. Second, the rankings reinforce a view of education as strictly instrumental to extrinsic goals such as prestige or wealth. Third, rankings create powerful incentives to manipulate data and distort institutional behavior for the sole or primary purpose of inflating one’s score. Because the rankings depend heavily on unaudited, self-reported data, there is no way to en sure either the accuracy of the information or the reliability of the resulting rankings.”

Source: Diver, Colin. “Is There Life After Rankings?” The Atlantic. November 2005.

23

Integrity continued…

“Freedom from temptation to game the ratings formula…

“Since the mid-1990s numerous stories in the popular press have documented how various schools distort their standard operating procedures, creatively interpret survey instructions, or boldly misreport information in order to raise their rankings.”

Source: Colin Diver (President, Reed College), “Is There Life After Rankings?” The Atlantic, November, 2005.

24

1. Failure to report low SAT scores from foreign students, athletes, other special admissions;

2. Exaggerate per capita instructional expenditure;3. Artificially drive up numbers of applicants;4. Inflate yield rate by rejecting or wait-listing the

highest achievers;5. Inflate graduate employment rate (law);6. Raise student selectivity (admit fewer 1st year

students (law);7. “Dumping” closest peers8. Self-promotion.



25


1. Focus more on grades and less on undergraduate institutions at admission.

2. Focus on LSAT.3. Other admission changes:

Admit fewer 1st year stud. more transfers. Reject some students with high LSAT. Reject students with limited prospects of employment. Focus scholarship of applicants with LSAT just above

median.4. Start part-time program.5. Focus the curriculum on what is needed for Bar

passage.

26

6. Spend money on glossy, colorful advertising.7. Raise tuition for all, increase scholarships to

those with numbers (LSAT, UGPA)8. Pay your own utilities.9. Encourage everyone and her sister to apply.10. Hire your own.11. Make it difficult for faculty members to leave in

the Fall.12. Increase the number of “books” in the “library”.13. Decrease funding of the library and other

unites with an abnormally high proportion of positive externalities.

Source: The Interplay between Law School Rankings, Reputations, and Resource Allocation: Ways Rankings Mislead. Jeffrey Evans Stake, Indiana Law Journal vol. 81, p. 229 (2006).

27

Integrity continued… “The problem is that the U.S. News college rankings are far from

reliable. Turns out that some of their numbers are made up.I know that first hand….

I was recently informed by the director of the data research at U.S. News, the person at the magazine who has a lot to say about how the rankings are computed, that absent students’ SAT scores, the magazine will calculate the college’s ranking by assuming an arbitrary average SAT score of one standard deviation (roughly 200 points) below the average score of our peer group…

The message is clear. Unless we are willing to be badly misrepresented, we had better send the information the magazine wants.”

Source: Michele Tolela Myers, President – Sarah Lawrence College. “The Cost of Bucking College Rankings.” The Washington Post, March 11, 2007.

28

Integrity continued …

“We conducted interviews with top management team members from the top 50 business schools in the USA to assess the effects of business school rankings on the conduct of business education. These informants characterized the rankings process predominantly as a game where the players face a field that is not always level and where the rules are not only ill-specified but also subtly changing.”

Kevin Corley & Dennis Gioia. (2000). “The Rankings Game: Managing Business School Reputation.” Corporate Reputation Review, 3(4).

29

The Israeli Higher Education System and

Quality Assessment

30

Israel: Some Basic Data – 2009

Area 22,145 sq. km. Population 7.5 million GDP ~780 billion NIS State Budget 316.5 billion NIS Education Budget 30.3 billion NIS Higher Ed. Budget 6.5 billion NIS

Not including Higher Ed. Budget

*

*

31

Higher Education in Israel Facts & Figures 2009/10

INSTITUTIONS 66 Universities 7 Open University 1 Art Academies 2 Comprehensive Colleges 12 Engineering Colleges 7 Teacher’s Colleges 24 Non-Budgeted Colleges 13

32

Higher Education in IsraelFacts & Figures

Students 280,000

Bachelor 221,420

Master 47,300

Ph.D. 10,300

Other (Dip.) 980

Faculty ~ 13,000

Tech & Admin. ~ 12,000

BUDGET (Tot.) ~$2 billion

33

Higher Education in Israel: Governance

Some 60% - 70% of the higher education budget comes from the Government

It is usually based on a 5-year plan

through … Negotiations between the Finance Ministry and the

Planning and Budgeting Committee (PBC) of the Council for Higher Education (CHE)

34

Breakdown of the Income of the Institutions of Higher

Education

PBC Allocations – 65%PBC Allocations – 65%

Tuition Fees – 21%Tuition Fees – 21%Other – 14%Other – 14%

35

The Council for Higher Education

The Law The framework of the system of higher

education in Israel is defined in the Council for Higher Education Law – 1958, with 11 amendments enacted over a period of 40 years.

This law established the Council for Higher Education and the procedures for the accreditation of institutions of higher education.

36

Article 15 of the Law guarantees that the

institutions of higher education are

autonomous in the conduct of their academic

and administrative affairs within the framework

of their budgets and their terms of

accreditation.

Academic Freedom

37

To grant a permit for the opening and maintenance of an institution of higher education;

To accredit an institution as an institution of higher education;

To revoke the accreditation of an accredited institution.

The Council’s ResponsibilitiesAccreditation

38

To authorize an accredited institution to

confer an academic degree

To approve new programs of study in

existing institutions

Approval of New Degrees & Programs

39

To license the branches and extensions of

foreign institutions of higher education which

operate in Israel.

Licensing Foreign Institutions

40

The Council delegated to the Planning and

Budgeting Committee (PBC) its responsibilities of

planning and budgeting.

The PBC is therefore the executive arm of the

Council.

The Planning and Budgeting Committee

41

To be an independent intermediary body between the Government and the institutions of higher education, in all matters relating to allocations for higher education

To negotiate with the Ministry of Finance the share of higher education in the state budget.

The PBC as a Buffer

42

To exclusively allocate the budget to

institutions of higher education, taking into

account the needs of society and the state,

while safe-guarding academic freedom and

assuring advancement of research and

teaching

Allocation of Funds

43

To ensure that institutional budgets are

balanced and executed according to plan

Accountability

To draw up plans for coordinated and efficient

development of higher education on the

national level

Planning and Coordination

44

To submit its recommendations to the

Council for Higher Education concerning

requests to open new institutions or new

units in existing institutions, after

examination of the planning and budgetary

points of view

Recommendations to the Council

45

The CHE Mandate: Summary

TERMS MENTIONED Securing funds Planning Licensing Accreditation Allocation of funds Accountability

TERMS ABSENT Review Re-accreditation Quality assurance Evaluation Assessment

46

CHANGE…June 2003 The CHE adopts the recommendation of a

National Committee to institute Quality Assessment and Assurance throughout the entire Higher Education System

2003/04 CHE establishes a QA unit and the first two

disciplines are chosen for a pilot evaluation

2004/05 The Process is underway…

47

What prompted this change? Some major reasons

Transition to mass higher education Internationalization of higher education Economic/budgetary pressure Pressure from stakeholders An inducive / ripe environment Perhaps . . . a realization by CHE that as part of

the expanded accreditation some control may have been lost and another look may be beneficial

48

Transition to Mass Higher Education

Institutions 1990/91 2005/06

Universities 7 7

Open University 1 1

Art Academies 2 2

Comprehensive Colleges 0 8

Engineering Colleges 2 8

Teachers’ Colleges 7 27

Non-Budgeted Colleges 2 8

Total 21 61

*Students 89,000 ~250,000

*Not including branches/operations of foreign institutions

49

Economic Budgetary Pressure

Government budgetary cuts Higher education institutions find themselves in

the red. Some blame: Lack of managerial-ism Lack of prioritization Lack of control and accountability

50

Pressure from Stakeholders

Government/ Politicians Boards of Trustees International Academic Advisory Committees Students Donors International Environment – General – Academic Industry (“Clients”)

51

Inducive /Ripe Environment

The 80s and 90s brought “In Search of Excellence” Deming et al. “Quality is Free” TQM

All Sectors – Industry, Public, Defense – become heavily involved with Quality

…Finally Higher Education joins in!

52

CHE - Realization

Accreditation

Re-accreditation

53

Main Purposes of the QA Activity

To create a culture of continuous quality improvement.

To bring about the continuous improvement of the various academic fields.

To be an active participant in the global HE quality evaluation and improvement endeavors .

54

Issues evaluated Mission, goals Study programs – all degrees Faculty – achievement, promotion criteria, etc. Students ─ admissions, grading, services, etc. Organization ─ committees; decision process Infrastructure ─ labs, library, IT, etc. Community involvement and cooperation

55

Main Features of Adopted QA Process

In developing our quality assessment process we “borrowed”

from the experience of many countries- Europe, USA,

Canada, Australia..: All Institutions are evaluated every 8 years (Not yet

implemented) All programs are evaluated every 6 years External Review Committee (top in discipline)

Appointed by & Reports to CHE On-Site visits by Committee

Self-evaluation process as basis for review.

56

Unique Features

All programs within a discipline are reviewed at the same time by the same committee.

Committee asked to assess “fitness for purpose”. No comparisons. No ranking.

Committee is asked to provide: Individual reports for each program to serve as guidelines

for improvement General overview of discipline to serve as a guideline for

CHE & PBC policy decisions Set of standards.

57

Trapped…

The decision to evaluate all programs in a given discipline ended up being an “invitation” for pressure to rank the evaluated programs.

CHE is for the time being withstanding this pressure claiming that ranking will inhibit the improvement process via all the “ranking games” possibilities which contradict the “fitness for purpose” approach.

58

Conclusions

Rankings of higher ed institutions – be it global or national – are probably here to stay. One cannot perceive a situation that will cause U.S. News & World Report, the Times Higher Ed. Supplement, Business Week, etc. to give up such a marketing bonanza. The question is to what extent should academia be a willingly contributing partner in this enterprising endeavor.

59

Conclusions continued…

If you don’t have a national ranking system try to delay.

If there is a ranking system try to live with it without losing sight of your own mission!

Or in the words of Colin Diver: “The Rankings are merely intolerable; unilateral disarmament is suicide.”


60

Back to Habermas…

Rankings seem to be detrimental to the social and cultural roles of the university and possibly to its overall mission.

61

THANK YOUTHANK YOU FOR FOR

LISTENINGLISTENING

Documents

1 “University Values – University Integrity” Integrity in Higher Education: Quality Improvement versus Ranking Improvement Nachum Finger Ben-Gurion University