Report of the Provost’s Faculty Task Force on Research Spring 2016 7/28/2016 Baruch College 2016 Faculty Task Force on Research

1

Executive Summary

Faculty Task Force on Research, Spring 2016

The Provost’s Faculty Task Force on Research met eight times during the 2016 spring semester and made the following recommendations.

I. The task force recommends widely promulgating the College P&B Guidelines of 2007 and basing the assessment of research and creative activity for personnel actions on those guidelines.

II. The task force recommends the development and promulgation of departmental “tie-in” guidelines on research and creative activity. Those guidelines would be qualitative and, if appropriate, they could also offer guidance on target journals and on impact metrics and other measures of scholarly quality.

III. The task force recommends that the assessment of research and creative activity by governance bodies and the candidate’s own assessment of those activities should consistently and transparently refer to the departmental “tie-in” guidelines and the College P&B Guidelines of 2007.

IV. The task force recommends that the Office of the Provost, the offices of the deans, and departments should communicate clearly about appropriate and inappropriate venues of publication, and particularly about venues of suspect merit, such as those identified in “Beall’s List.”

V. The task force recommends that departments establish guidelines on appropriate and valued forms of digital research and creative activity, and on other emerging modes of scholarly and creative communication, presentation, and representation.

VI. The task force details concrete recommendations that would help to promote the culture of research and creative activity at Baruch College. Those recommendations would assist in fostering and supporting communities of scholars and artists among faculty and students. The following report also details ways of recognizing and valorizing those activities.

2

Introduction

In spring 2016, the Provost and Senior Vice President for Academic Affairs, David Christy, invited the following faculty and staff members to serve on a Faculty Task Force on Research, chaired by Associate Provost Erec R. Koch.

Professor Katherine Behar, Department of Fine and Performing Arts, Weissman School of Arts & Sciences Professor John Brenkman, Department of English, Weissman School of Arts & Sciences Vice President and Dean Arthur Downing, Information Services and the Library Dominic Esposito, Director, Sponsored Programs and Research Erec R. Koch, Associate Provost for Academic Administration and Faculty Development Professor Sanders Korenman, School of Public and International Affairs Professor Lin Peng, Department of Economics and Finance, Zicklin School of Business Professor Kristin Sommer, Department of Psychology, Weissman School of Arts & Sciences Professor Ramona Zachary, Narendra Paul Loomba Department of Management, Zicklin School of Business

The Provost charged the task force with recommending effective ways to assess faculty research and creative activity, especially for new and emerging modes of scholarly communication, and ways to promote the culture of research and creative activity at Baruch College. The task force met eight times during spring 2016; its recommendations follow.

3

Task Force Recommendations To advance the research mission of the College, the task force recommends that departments, P&B committees, and academic administrators give priority to three goals: To foster research productivity at all ranks;

To clarify expectations and enhance the prospect of their realization by faculty who are pursuing tenure and promotion or promotion after tenure;

To adopt and embrace discipline-specific or department-specific guidelines for the evaluation of the qualitative and quantitative productivity of faculty undergoing review in the tenure and/or promotion processes.

Recommendation I: The College P&B Guidelines of 2007

The College P&B Guidelines of 2007, scholarship section (see Appendix IV), are appropriate and clear guidelines for personnel actions, including tenure and promotion. The task force recommends:

1) Wide promulgation and greater visibility of those guidelines. Moreover, those guidelines should be used in assessments of candidates’ progress toward and readiness for tenure and promotion at all levels of review and evaluation (chairs’ reports, departmental executive committees/tenure committees, school P&Bs, and College P&B). Concurrently, chairs and deans should present those guidelines to new faculty upon appointment, and faculty should use them in self-assessments of progress toward tenure and promotion: namely, in their annual personal (candidates’) statements and the research plans embedded in those statements.

2) Governance bodies should use those guidelines as a basis for recommendations on reappointment, tenure, and promotion, especially in the category of research/creative activity.

3) The task force recommends that the President confirm that her/his decisions on tenure and promotion are based on those guidelines, especially as concerns the category of scholarship.

Recommendation II: Departmental “Tie-in” Guidelines on Research and Creative Activity

Departments are encouraged to develop “tie-in” guidelines on research and creative activity that emphasize peer-review, appropriateness of venue, impact on and contribution to the discipline, and ongoing research activity and that are consistent with and offer disciplinary specificity to the College P&B guidelines. Those guidelines should be communicated to all governance bodies that are involved in personnel actions.

1) Guidelines should take the form of a qualitative statement such as “evidence of ongoing research activity as represented by regular peer-reviewed publications that are of significant length, are

4

placed in appropriate venues, and make valuable contributions to the candidate’s discipline”; or “evidence of ongoing creative practice as represented by exhibition, performance, or artistic presentation of peer-reviewed works of significant depth, scope, and value that are placed in appropriate venues and that make significant contributions to the candidate’s discipline.” Departments may want to offer recent examples of successful reappointment, tenure, and promotion, profiles.

2) As part of those guidelines, a department may offer specific guidance on the ways in which that unit determines the value of contributions in research and creative activity. Those valuations will be specific to a discipline but must be applicable to all candidates in that discipline. Such guidance may include:

a. Lists or examples of peer-reviewed journals, and the basis of those recommendations; b. Lists of predatory or inappropriate journals to exclude as venues; c. Metrics and measures of impact that are appropriate to a discipline. (See Appendix I.)

The task force recommends that departments that endorse the use of research metrics frame such determinations within the 10 principles of “The Leiden Manifesto for Research Metrics” (http://www.nature.com/news/bibliometrics-the-leiden-manifesto-for-research-metrics-1.17351 and Appendix III). The task force particularly values principle 7, which states the imperative that one “base assessment of individual researchers on qualitative judgment of their portfolio.” The task force recommends that metrics should not be used as a stand-alone, objective judgment of the value of a research contribution; however, metrics may be one component of or may inform assessments.

Recommendation III: Assessment of Research

At all levels of review, governance bodies should strive to determine:

1) The appropriateness and value of a venue of publication or creative activity; 2) The importance and the significance of the contribution to research and creative activity

in the discipline or field; 3) If appropriate, how the research or creative activity contributes to a community of

researchers or artists in a department or school or within a discipline.

Candidates’ statements should also address, in light of departmental guidelines, the issues raised immediately above. The candidate should explain:

1) The appropriateness and value of a venue of publication or creative activity; 2) The importance and the significance of the contribution to research and creative activity

in the discipline or field; 3) If appropriate, how the research or creative activity contributes to a community of

researchers or artists in a department or school or within a discipline.

Moreover, those points should be an ongoing part of the candidate’s annual reviews with the department chair and should inform the evaluations of and reports for the candidate.

5

Candidates’ statements should present a research plan annually so that part of their assessment and self-assessment in subsequent years will include their progress toward goals previously set or explanations of modifications of goals, as necessary.

Governance bodies and candidates should make use of research metrics as a means of assessment in light of the comments made above in Recommendation II.2 above.

Recommendation IV: Predatory Publishers

The task force endorses using “Beall’s List of Predatory Publishers 2015 [2016]” https://scholarlyoa.com/2015/01/02/bealls-list-of-predatory-publishers-2015/ , as the principal College resource for identifying predatory publishers and journals.

The task force invites schools and departments to provide supplementary lists and/or warnings about predatory publishers or journals, or other journals in which an academic unit recommends not publishing.

The task force recommends that the Baruch College Provost communicate annually to the faculty about appropriate and inappropriate publication venues, particularly venues of suspect merit such as those identified in “Beall’s List.”

Recommendation V: New Modes of Scholarly Communication, Presentation, and Representation

Baruch College faculty have begun to publish research and present creative activity on new platforms, especially digital platforms. Those digital platforms include new modes of scholarly communication—from blogs to more “traditional” electronic journal articles and books--, web-based data bases, and web-based multi-media presentations. The task force recognizes that a decision on whether or not a particular type of research or creative activity has disciplinary merit should be made by the faculty of individual departments, in consultation with the correlative dean’s office and the Office of the Provost. The task force read several documents prepared by professional associations and recommends that departments refer to those and other pertinent resources in assessing the value of new modes of scholarly and creative communication, presentation, and representation.

1) The Modern Language Association, Guidelines for Evaluating Work in Digital Humanities and Digital Media https://www.mla.org/About-Us/Governance/Committees/Committee-Listings/Professional-Issues/Committee-on-Information-Technology/Guidelines-for-Evaluating-Work-in-Digital-Humanities-and-Digital-Media

2) The College Art Association (CAA), Case Studies and Examples for Evaluating Digital Scholarship http://www.collegeart.org/news/2016/02/23/case-studies-and-examples-for-evaluating-digital-scholarship/

3) American Historical Association, Suggested Guidelines for Evaluating Digital Media Activities in Tenure, Review, and Promotion: An AAHC Document http://www.historians.org/publications-and-directories/perspectives-on-history/october-2001/suggested-guidelines-for-evaluating-digital-media-activities-in-tenure-review-and-promotion-an-aahc-document

6

See Appendix II for an example of the inclusion of digital scholarship and creative activity in departmental standards for reappointment, tenure, and promotion.

Each department should create a statement about appropriate and valuable types of digital products of research and creative activity, while keeping in mind that rigorous peer-review of such products is essential to establishing their scholarly value. Such statements should be communicated to all governance bodies involved in personnel actions. Departments should also review and update those statements on a regular basis, as new digital products of scholarly and creative activity emerge. The individual faculty author or creator shares responsibility for the establishment of the scholarly or artistic value of such products.1

We agree with CAA Guidelines that state that “meritorious work published in open access venues should not be discounted because it does not travel through traditional distribution channels” (CAA Guidelines).

In tenure and promotion cases, external evaluation is a key component in assessing a candidate’s contributions in research and creative activity. In cases where the digital platform is itself part of the candidate’s research or creative activity, departments should consider content specialists with some expertise in that medium or an additional expert specialized in that medium.2

In cases where review of the scholarly product depends on the medium of its delivery, we urge departments, governance bodies, and external evaluators to conduct their assessments in that medium. For example, the MLA Guidelines state the following:

Respect Medium Specificity When Reviewing Work. Since scholarly work is always designed for presentation in a specific medium, evaluative bodies should foreground medium specificity by reviewing faculty members’ work in the medium for which it was produced. For example, born-digital and Web-based projects are often spatial, interactive, iterative, and networked. If possible,

1 We endorse the following pertinent statements from MLA and CCA guidelines. “Delineate and Communicate Responsibilities. When chairs and hiring committees seek candidates who have expertise in the use and creation of digital media, explicit reference to such work should be included in job descriptions, and candidates should be apprised of their responsibilities relative to this work. When candidates wish to have digital work considered an integral part of their positions, the department should make clear to candidates at the time of hiring its expectations for such work and for the candidate’s productivity, its responsibilities in supporting such work, and how it plans to give recognition to the work.” (MLA Guidelines) “In the research statement, scholars should make evident which uses of digital technology mark the significant contributions of their work and why these technologies are appropriate for a particular art or architectural question While the primary responsibility for clarifying the scholarly contribution of the use of digital technologies rests with the scholar, departments and colleges/universities must articulate appropriate categories of digital scholarship in their evaluation documents as well as what clarification they expect from a scholar under consideration. This may, for example, take the form of a separate brief document explaining the use of the digital technology to non-specialists.” (CAA Guidelines) 2 “Engage Qualified Reviewers. Faculty members who work in digital media or digital humanities should be evaluated by persons practiced in the interpretation and development of new forms and who are knowledgeable about the use and creation of digital media in a given faculty member’s field. At times this may be possible only by engaging qualified reviewers from other departments, divisions, or institutions. If faculty members worked collaboratively with colleagues from other disciplines, then departments and institutions should seek the assistance of experts in those other disciplines to assess and evaluate the work.” (MLA Guidelines) “In peer review, the evaluation should include both a content expert and a digital expert, or one person who is both.” (CAA Guidelines)

7

they should be viewed in electronic form, not in print or as snapshots of dynamic behavior. (MLA Guidelines)

Recommendation VI: Promoting a Culture of Research and Creative Activity at Baruch College.

The task force is committed to enhancing the imbrication of research and creative activity with teaching, which should mutually inform one another. Research is a vital part of faculty identity, and shared areas of intellectual interest and research synergies foster a sense of academic community for all.

The task force endorses existing fora that provide opportunities to present research, completed or in progress, and to get constructive feedback from colleagues. We especially support such initiatives that are sustained over the course of a semester, an academic year, or longer. The Office of the Provost sponsors and funds Faculty Research Seminars, which bring together Baruch College faculty, graduate students, advanced undergraduates, faculty from neighboring institutions, and visiting scholars in interdisciplinary or disciplinary groups for the purpose of exploring topics of common intellectual and scholarly concern. Schools also sponsor similar series for presenting work in progress or completed research, such as the weekly SPA Faculty Research Seminar. The Weissman School of Arts and Sciences’ monthly Global Studies Seminar is a vibrant forum and incubator for faculty research and curriculum development in global studies and education. The Weissman-based “U.S-Europe Seminar: Ideas / Politics / Literature / Economy” provides a highly interactive forum for discussion among faculty from Baruch and other New York-area institutions, policy analysts, journalists, and publishers on a wide range of topics.

A number of departments sponsor seminars and works-in-progress series. The Department of English, Weissman School of Arts and Sciences, sponsors a faculty work-in-progress lecture twice a semester. The Narendra Paul Loomba Department of Management, Zicklin School of Business, runs two faculty research seminars: the Delta Research Seminar Series, which features talks in the different disciplines in management, including strategic management, business and society, entrepreneurship, and organizational behavior, and the Omega Seminar Series in Operations and Supply Chain Management, which utilize various operations research, optimization and/or economics-based theories (such as game theory) with business applications in retail, supply chain, healthcare and service industries. The Bert W. Wasserman Department of Economics and Finance offers two regular weekly seminar series, one built around external speakers, and one featuring works-in-progress presentations by faculty and PhD candidates. That department also hosts a PhD Research Conference each semester where PhD students present their work, which is discussed by faculty and students.

The task force recommends the development of more seminars and research groups of this kind so that every faculty member will have a research community and “home” within the College. Schools may want to consider, for example, hosting short-term seminars or symposia on topics of broad interest to many faculty. The task force would welcome the participation of school and College administrators as presenters or interlocutors in those seminars. The task force also recommends the dedication of funding at the department, school, and College levels—ideally through a matching funds system at those many levels. The Office of the Provost and the school deans’ offices may want to consider issuing an annual RFP to identify seminars that can be co-funded.

8

The task force recommends the development of a “self-service” clearinghouse for announcing seminars and seminar meetings. This can be achieved through the existing college calendar or through an interactive module that could be filled out and submitted by the seminar organizers.

The task force encourages the participation of advanced undergraduates and graduate students in these initiatives, as a way to promote research as an integral part of their education. These efforts would also align with initiatives such as Critical Inquiry Day.

The task force recognizes that many research initiatives target untenured faculty. Untenured faculty are eligible for CBA reassigned time for the compelling reason that they must launch their research agendas. The Eugene M. Lang fellowship similarly supports the research efforts of untenured faculty. The task force fully endorses these efforts that promote research by untenured colleagues, but it also would like to encourage support for tenured faculty. Some such initiatives already exist, such as the Baruch College Grant Development Program, which offers a course release—and consequently time—to faculty who are preparing the submission of a significant grant or fellowship. The task force would welcome a program similar to the Lang Fellowship that would support the research of tenured associate and full professors.

The task for also recommends the development of fora that foreground faculty as researchers from the time of their arrival at Baruch. One possibility would be to host an evening of research-focused PechaKucha early in the fall semester for faculty newly appointed to the College.

The task force recommends the development of meaningful forms of recognition of faculty accomplishments. The increased presence of faculty news in College communications and the development of special events such as the receptions for faculty authors and for the presentation of the president’s excellence awards are welcome and important. The task force would also encourage the development of financial rewards for such accomplishments. Those modes of recognition could include summer research awards, short-term endowed or named chairs that carried a salary supplement and/or research funds, and a College-based multi-tiered salary system or more extensive use of the Salary Above Base system.

9

Appendix I: Research Metrics

The following is a list of resources that departments may consult as they consider the efficacy and value of research metrics in their disciplines. All of the links below provide measures of research impact that may be pertinent to a particular discipline.

Web of Knowledge - Journal Citation Reports (http://guides.newman.baruch.cuny.edu/databases/I-J and select “Journal Citation Reports”) SCImago (http://www.scimagojr.com/) Google Scholar (http://scholar.google.com and select “Metrics”) Social Science Research Network (http://ssrn.com) Faculty should also be alerted to the existence of “predatory” impact indices. The example given immediately below is of and impact index created by a predatory publisher. https://sites.google.com/site/photonfoundationorganization/home/impact-index

10

Appendix II: Sample of Research Guidelines for Tenure and Promotion That Include Digital Scholarship and Creative Activity

See bold italics below.

(Available from https://www.uky.edu/Faculty/Senate/files/Meetings/1_2013-2014/20140210/WRD%20Department%20Proposal%20Master%20File%201-27-14_TOUS.pdf , “Proposal for a Department of Writing, Rhetoric, and Digital Studies,” University of Kentucky, p.17 of download)

Section 2: Guidelines for Appointment, Review, Promotion and Tenure: Professors A. Receipt of Tenure Faculty members earning tenure in the Department will have contributed to the mission and needs of the department. To achieve tenure, they must demonstrate a significant, sustained contribution to their discipline, Department, College, and University. Assistant professors earning tenure will have a strong record of successful (1) research and publication and (2) teaching, as well as an appropriate level of (3) service activity. Candidates seeking tenure will provide evidence of achievement in each of these three basic areas of performance. Substantial achievement in both research and teaching, and indications of future excellence in these categories, are necessary for tenure; because of the mission of WRD, local citizenship is also expected. An outstanding record in any one of these three categories alone is insufficient for tenure. 1. Research. From the time of appointment, candidates for tenure must demonstrate a strong commitment to research and publication in the areas of writing studies, rhetoric studies, and/or digital media studies. We also recognize scholarly work presented in digital media (e.g., video production, digital installations) as contributing to WRD’s mission under the category of research/creative activity. Because of the nature of the field, scholarship that focuses on learning and pedagogy (especially of writing) and results in traditional categories of research (peer-reviewed publications, conference presentations, and invited lectures and workshops) is valued the same as other research in similar categories. Also relevant are substantial external grants for projects that will contribute to knowledge in the field. Collaborative authorship is recognized as an established practice for scholarship in our field, and on-line publications are recognized as important and will be judged on their merit (e.g., peer-review, low acceptance rates, and level of prestige). It is important to emphasize that publications in new outlets count the same as publications in established ones. Finally, the Department rewards innovative uses of new media and recognizes that some important work is not peer-reviewed in the usual ways. In such cases, ex post reviews of such work by highly regarded scholars chosen by the Department can be submitted as evidence of merit. Scholarly publications or activities include but are not necessarily limited to the following: Book-length publications, including traditional academic or creative publications, textbooks, co-authored work, translations, special Proposal for a Department of Writing, Rhetoric, and Digital Media

11

18 editions) Digital projects Databases Peer-reviewed articles in print or online Other articles or essays Book reviews Grants Editing (special issues, etc.) Visiting lectures at other universities Keynote addresses (local or national) Peer-reviewed conference presentations Other conference presentations Workshops or other special events University presentations Peer reviews (of grants and other submissions) Editorial boards Other work on professional organizations in support of original research and creative activity Tenure and promotion in WRD are awarded to faculty who produce significant scholarship that represents a cohesive, long-term scholarly project or agenda. Typically, candidates for tenure will be expected to present to reviewers evidence of sustained, scholarly achievement in one of two forms: (1) a book published in a reputable scholarly press or (2) a series of scholarly articles in peer-reviewed journals, along with other evidences. The Department also recognizes major digital work that represents substantial, original research as a possible centerpiece of a promotion and tenure case. Whether a faculty member presents a scholarly monograph, a collaborative book, a series of published articles, or a digital project as the major evidence of her or his scholarly achievement, that work must be peer-reviewed and the candidate should be able to demonstrate its value to the broader discipline and/or to trans-disciplinary conversations connected to rhetoric and composition. Junior faculty pursuing digital scholarship as the centerpiece of their tenure and promotion case must be able to demonstrate the value of their projects as scholarship, no matter what the venue or medium. In the case of digital scholarship, candidates will be required to demonstrate the value of the venues and media in which they publish and that the scope and rigor of the project(s) are comparable to that of a book or series of articles. Engagement with scholarship in the faculty member’s field or subdisciplinary areas, scope and/or originality of the scholarly argument, and innovative scholarly methods are among the ways a candidate can demonstrate the scholarly value of the work. The Department recognizes co-authored books as equally viable evidence of scholarly achievement when (1) the book exhibits the scope, rigor, and Proposal for a Department of Writing, Rhetoric, and Digital Media 19 quality that outstanding single authored books, and (2) when the candidate can clearly present his or her role in the collaboration.

12

Appendix III. The Leiden Manifesto

10 principles of “The Leiden Manifesto for Research Metrics” (detailed at http://www.nature.com/news/bibliometrics-the-leiden-manifesto-for-research-metrics-1.17351 )

1) Quantitative evaluation should support qualitative, expert assessment. Quantitative metrics can challenge bias tendencies in peer review and facilitate deliberation. This should strengthen peer review, because making judgements about colleagues is difficult without a range of relevant information. However, assessors must not be tempted to cede decision-making to the numbers. Indicators must not substitute for informed judgement. Everyone retains responsibility for their assessments.

2) Measure performance against the research missions of the institution, group or researcher. Programme goals should be stated at the start, and the indicators used to evaluate performance should relate clearly to those goals. The choice of indicators, and the ways in which they are used, should take into account the wider socio-economic and cultural contexts. Scientists have diverse research missions. Research that advances the frontiers of academic knowledge differs from research that is focused on delivering solutions to societal problems. Review may be based on merits relevant to policy, industry or the public rather than on academic ideas of excellence. No single evaluation model applies to all contexts.

3) Protect excellence in locally relevant research. In many parts of the world, research excellence is equated with English- language publication. Spanish law, for example, states the desirability of Spanish scholars publishing in high-impact journals. The impact factor is calculated for journals indexed in the US-based and still mostly English-language Web of Science. These biases are particularly problematic in the social sciences and humanities, in which research is more regionally and nationally engaged. Many other fields have a national or regional dimension — for instance, HIV epidemiology in sub-Saharan Africa. This pluralism and societal relevance tends to be suppressed to create papers of interest to the gatekeepers of high impact: English-language journals. The Spanish sociologists that are highly cited in the Web of Science have worked on abstract models or study US data. Lost is the specificity of sociologists in high-impact Spanish- language papers: topics such as local labour law, family health care for the elderly or immigrant employment. Metrics built on high-quality non-English literature would serve to identify and reward excellence in locally relevant research.

4) Keep data collection and analytical processes open, transparent and simple. The construction of the databases required for evaluation should follow clearly stated rules, set before the research has been completed. This was common practice among the academic and commercial groups that built bibliometric evaluation methodology over several decades. Those groups referenced protocols published in the peer- reviewed literature. This transparency enabled scrutiny. For example, in 2010, public debate on the technical properties of an important indicator used by one of our groups (the Centre for Science and Technology Studies at Leiden University in the Netherlands) led to a revision in the calculation of this indicator. Recent commercial entrants should be held to the same standards; no one should accept a black-box evaluation machine. Simplicity is a virtue in an indicator because it enhances transparency. But simplistic metrics can distort the record (see principle 7). Evaluators must strive for balance — simple indicators true to the complexity of the research process.

5) Allow those evaluated to verify data and analysis. To ensure data quality, all researchers included in bibliometric studies should be able to check that their outputs have been correctly identified. Everyone directing and managing evaluation processes should assure data accuracy, through self-verification or third-party audit. Universities could implement this in

13

their research information systems and it should be a guiding principle in the selection of providers of these systems. Accurate, high-quality data take time and money to collate and process. Budget for it.

6) Account for variation by field in publication and citation practices. Best practice is to select a suite of possible indicators and allow fields to choose among them. A few years ago, a European group of historians received a relatively low rating in a national peer-review assessment because they wrote books rather than articles in journals indexed by the Web of Science. The historians had the misfortune to be part of a psychology department. Historians and social scientists require books and national- language literature to be included in their publication counts; computer scientists require conference papers be counted. Citation rates vary by field: top-ranked journals in mathematics have impact factors of around 3; top-ranked journals in cell biology have impact factors of about 30. Normalized indicators are required, and the most robust normalization method is based on percentiles: each paper is weighted on the basis of the percentile to which it belongs in the citation distribution of its field (the top 1%, 10% or 20%, for example). A single highly cited publication slightly improves the position of a university in a ranking that is based on percentile indicators, but may propel the university from the middle to the top of a ranking built on citation averages.

7) Base assessment of individual researchers on a qualitative judgement of their portfolio. The older you are, the higher your h-index, even in the absence of new papers. The h-index varies by field: life scientists top out at 200; physicists at 100 and social scientists at 20–30 (ref. 8). It is database dependent: there are researchers in computer science who have an h-index of around 10 in the Web of Science but of 20–30 in Google Scholar. Reading and judging a researcher’s work is much more appropriate than relying on one number. Even when comparing large numbers of researchers, an approach that considers more information about an individual’s expertise, experience, activities and influence is best.

8) Avoid misplaced concreteness and false precision. Science and technology indicators are prone to conceptual ambiguity and uncertainty and require strong assumptions that are not universally accepted. The meaning of citation counts, for example, has long been debated. Thus, best practice uses multiple indicators to provide a more robust and pluralistic picture. If uncertainty and error can be quantified, for instance using error bars, this information should accompany published indicator values. If this is not possible, indicator producers should at least avoid false precision. For example, the journal impact factor is published to three decimal places to avoid ties. However, given the conceptual ambiguity and random variability of citation counts, it makes no sense to distinguish between journals on the basis of very small impact factor differences. Avoid false precision: only one decimal is warranted.

9) Recognize the systemic effects of assessment and indicators. Indicators change the system through the incentives they establish. These effects should be anticipated. This means that a suite of indicators is always preferable — a single one will invite gaming and goal dis- placement (in which the measurement becomes the goal). For example, in the 1990s, Australia funded university research using a formula based largely on the number of papers published by an institute. Universities could calculate the ‘value’ of a paper in a refereed journal; in 2000, it was Aus $800 (around US $480 in 2000) in research funding. Predictably, the number of papers published by Australian researchers went up, but they were in less-cited journals, suggesting that article quality fell.

10) Scrutinize indicators regularly and update them. Research missions and the goals of assessment shift and the research system itself co-evolves. Once-useful metrics become inadequate; new ones emerge. Indicator systems have to be reviewed and perhaps modified. Realizing the effects of its simplistic formula, Australia in 2010 introduced its more complex Excellence in Research for Australia initiative, which emphasizes quality.

14

Appendix IV: College P&B Guidelines of 2007, Scholarship Section

15

16