Key Issues and Trends in Evaluating Research Excellence in Applied Development Contexts: A Review and Synthesis of the Serial and Grey Literature

Prepared by Pedro F. Mateu, Kristin A. Hobson, Chris L. S. Coryn, and Daniela C. Schröter. The Evaluation Center Western Michigan University Kalamazoo, MI Funded by the International Development Research Centre, Ottawa, Canada. Paper presented at 10th EES Biennial Conference, Helsinki, Finland, October 3 - 5, 2012. Keywords: research evaluation frameworks; research excellence evaluation; research excellence; research evaluation

Abstract The paper presents a summary of key issues and trends in evaluating research, based on a document review. The summary focuses on purpose of research excellence frameworks, existing frameworks and in which context they can be applied, and strengths and weaknesses of the existing frameworks.

Biography Pedro Mateu and Kristin A. Hobson are doctoral students and research associates in the Interdisciplinary PhD in Evaluation (IDPE) at Western Michigan University (WMU). Pedro specializes in evaluation of development projects and public services while Kristin’s evaluation interest centers on evaluation practice and evaluation design. Chris Coryn is the Director of the WMU IDPE. His research focuses on evaluation theory and practice, quantitative and qualitative research methods, measurement, and research synthesis and meta-analysis. Daniela Schröter is the Director of Research in WMU’s Evaluation Center. Her research interests center on evaluation practice, methodology, communication, and capacity building.

Introduction Not uncommon is it for funders to want to allocate available funds for research to the researchers and organizations with the most promise to deliver excellent research – whether it’s emergent philosophies and models, explanations for processes, or solutions to extant problems. Doing so, requires evaluating the researchers and/or research, not only through the “gold standard” of bibliometrics and peer review. As such, this report describes a review and synthesis of the serial and grey literature on key issues, trends, definitions, and approaches to evaluating research excellence in applied development contexts. The paper answers the following two broad questions:

1. How is research excellence defined in the literature? 2. What are the commonalities in purposes, uses, users, strengths, and

weaknesses across the existing frameworks and mechanisms that are used to evaluate research excellence?

Based on a set of 19 frameworks that were created for evaluating research excellence, the authors define research excellence as four sub-dimensions: research quality, outputs, impacts, and environment. The report also provides insight into the 19 frameworks; specifically, mechanisms for funding and evaluating research around the world. Lastly, the report introduces the reader to a changing world of research evaluation that was dominated by quantitative methods and where qualitative methods are becoming accepted as reliable. Evidently, evaluating research is neither white nor black; it is grey with a multitude of paths for evaluating research where none of them completely assure the existence of causation or attribution.

Methods

Data Sources Primary papers were searched for in Google Scholar and Google using search terms for research excellence (e.g., research quality, research impact, research environment, research output, research outcomes). Additional primary papers were collected through recommendations of articles to review as well as by reviewing the reference lists of retrieved studies.

Paper Selection Papers were identified and selected for analysis if they met one of the following inclusion criteria: (1) presented a framework or mechanism, or part of a framework or mechanism, for evaluating research excellence; or (2) critiqued or analyzed a framework or mechanism for evaluating research excellence. From the searches and application of the inclusion criteria, 52 papers, of which one was from the grey literature, were retrieved and coded into a database for analysis.

Data Extraction A standardized data extraction form (in Microsoft Access) was used to code information from the included papers. Through this process, the first two authors of this report extracted and recorded the following information for each paper: bibliographic information, synopsis of the paper, definition of research excellence, purpose of the framework, use and users of the framework, strengths and weaknesses of the framework, and relevance and appropriateness of the framework to the international context and IDRC.

Data Synthesis In analyzing and summarizing the data, the authors answered two broad questions. First, how is research excellence defined in the literature? Second, what are the commonalities in purposes, uses, users, strengths, and weaknesses across the existing frameworks and mechanisms that are used to evaluate research excellence?

Dimensions of Research Excellence

Definitions of research excellence vary widely. Even so, four consistent dimensions emerged from the literature review, including research quality, outputs, impacts, and environment.

Research Quality In terms of research excellence, research quality refers to epistemic, social, and economic robustness as well as the importance and consistency of research results. Epistemic robustness comprises scientific, methodological, and theoretical rigour as encompassed in the scientific process (Furlong & Oancea, 2005; Research Trends, 2008). Sub-dimensions of epistemic robustness include trustworthiness, builds on what is known and contribution to knowledge, explicitness, propriety, and paradigm-dependent criteria (Furlong & Oancea, 2005). Suggested measures include the number of studies, sample size, statistical power, and effect sizes (West, King, Carey, Lohr, McKoy, Sutton, & Lux, 2002; Research Trends, 2008). Social and economic robustness refers to technological indicators, capacity development, value for people, and economic indicators. Technological robustness includes purposivity, salience/timeliness, specificity, accessibility, concern for enabling impact, flexibility, and operationalizability. Capacity development and value for people involves plausibility, partnership, collaboration, engagement, reflexivity, deliberation, criticism, receptiveness, transformation, and personal growth (Furlong & Oancea, 2005). Economic indicators are marketability, competiveness, cost-effectiveness, auditability, feasibility, and originality (Furlong & Oancea, 2005) as well as the sustainability of research (Meek & Van der Lee, 2005).

Research Outputs Another consistently described dimension of research excellence is the quantification of “high quality” research outputs. Output indicators encompass volume, quality, impact, and utility (Geuna & Martin, 2003; Higher Education Funding Council for England, Scottish Funding Council, Higher Education Funding Council for Wales, & Department for Employment and Learning, 2009; SCImago’s Lab. 2011; Frontier Economics, 2009).

Research Impacts Resulting from research outputs are impacts. Research impact represents the social, cultural, environmental, and economic returns that research outputs produce for wider society, outside academia (Butler, 2008, Donovan, 2011; Nutley, Walter, & Davies, ,2007; Higher Education Funding Council for England, Scottish Funding Council, Higher Education Funding Council for Wales, & Department for Employment and Learning, 2009; Richardson, 2011).

Research Environment In regard to environment as a dimension of research excellence, the literature denotes that assessment of the environment involves determining the research infrastructure and supporting activity that contributes to dissemination and application of the research (Higher Education Funding Council for England, Scottish Funding Council, Higher Education Funding Council for Wales, & Department for Employment and Learning, 2009).

Purposes of Existing Frameworks The literature search yielded 19 unique frameworks and mechanisms that are sometimes used to evaluate the aforementioned dimensions of research excellence. One common purpose is the allocation of public funds to universities and research centers’ at a national level. The Australian Research Qualitative Framework (ARQF), German Excellence Initiative Strategy, United Kingdom Research Excellence Framework (REF) for higher education institutions and the Quality Assessment Mechanism of the United Kingdom Royal Academy of Engineering (RAENG), the Common Strategic Framework for European Union Research and Innovation Funding (Horizon 2020), and the Spanish National Evaluation and Foresight Agency (ANEP, Agencia Nacional de Evaluacion y Prospectiva), which belongs to the Spanish Research Funding System, have their own standards of research quality assessed primarily through a peer review process (Butler, 2008; Donovan, 2008; Higher Education Funding Council for England, Scottish Funding Council, Higher Education Funding Council for Wales, & Department for Employment and Learning, 2009; 2011; European Commission, 2011; Kleiner, 2011; The Royal Academy of Engineering, 2000; Vey, 2010). Ultimately, frameworks from Germany and the European Union look for increasing international

competitiveness and industrial leading markets (European Commission, 2011; Kleiner, 2011). The evaluation of research impacts is another common purpose between three frameworks: SIAMPI (Social Impact Assessment Methods for research and funding instruments through the study of Productive Interactions between science and society), Payback Framework, and Methodology for Evaluation of the Societal Impact of (applied) Health Research (Donovan, 2011; Royal Netherlands Academy of Arts and Sciences, 2002; Spaapen & van Drooge, 2011). In addition, the second and third frameworks have another commonality. Both measure the impacts of medical and health research. Elsevier’s Measure of Excellence and the SCImago’s Evaluation Framework of Research Performance are intended to measure research excellence for comparison purposes across different research organizations (i.e., universities, research centers, or other type of research institutions). The first is used to evaluate linkages between research excellence and research funding policy at the country level, which includes the United Kingdom, United States of America, China, Japan, Australia, and Germany (Wellings & Winzer, 2011). The second provides a ranking of 3,042 worldwide research organizations (SCImago Lab, 2011). The purpose of two of the frameworks (Qualitative Research Tool and Quantitative Research Tool) is to help readers evaluate the accuracy of research articles published on the Child Care and Early Education Research Connections website (Child Care & Early Education Research Connections, n.d.). Finally, a research assessment exercise called Valutazione Triennale della Ricerca (Italian VTR) is used to assess research productivity, from 2001 to 2003, by universities and other research institutions under the Ministry of Education, University, and Research funding in Italy (Franceschet, & Costantini, 2011; Reale, Barbara, & Costantini, 2007). In contrast to the analysis of the 19 frameworks, which identified five purposes of research evaluation, Molas-Gallart (2012) identified three purposes: distributive, improvement, and controlling. This classification was based on how the evaluation practice contributes to policy practice in terms of allocation of resources, conduction of activities to reach the policy goals, and implementation of controls on the use of allocated resources (Molas-Gallart, 2012). Using the fourth step of the general logic of evaluation – synthesizing and integrating data into a judgment of merit (Fournier, 1995) of researchers, research centers, organizations, or any evaluand related to research, – research evaluation accomplishes its distributive purpose by helping research funders to allocate their resources according to merits (Molas-Gallart, 2012). In all research processes, nothing is absolute. Most of the time, the expected activities are adjusted based on past lessons learned in order to improve performance. In this situation, research evaluation pursues an improvement purpose, which is supported by a feedback mechanism and the evaluand’s

willingness to learn by doing (Molas-Gallart, 2012). Finally, evaluation plays the controlling purpose when the research funders assess the inputs and outputs of the research activities (Molas-Gallart, 2012). The results of the (controlling) evaluation illuminate whether the resources allocated were used to conduct activities that achieved the researches’ objectives (Molas-Gallart, 2012). In regard to this controlling purpose, the Spanish National Research Plan has an evaluation tool called Integral Monitoring and Evaluation System, which is a monitoring instrument for public policies in research and development (Vey, 2010). The instrument is applied to evaluate research and innovation policies on a yearly basis and the results yielded are used to control the management of public funding for research and development (Vey, 2010).

Uses of Existing Frameworks The frameworks are predominately used to evaluate the processes of conducting research and the products of research. Seven frameworks were created for funding research in universities and research centers—the United Kingdom REF, Quality Assessment Mechanism of the United Kingdom RAENG, ARQF, the Spanish Research Funding System, German Excellence Initiative Strategy, Horizon 2020, and the Methodology for Evaluation of the Societal Impact of (applied) Health Research in the Netherlands. The first three frameworks evaluate research produced by researchers at higher education institutions (Higher Education Funding Council for England, Scottish Funding Council, Higher Education Funding Council for Wales, & Department for Employment and Learning, 2009; 2011; The Royal Academy of Engineering, 2000; Butler, 2008), including the Italian Valutazione Triennale della Ricercabut (Franceschet, & Costantini, 2011). The Spanish, German and European Union frameworks include not only research centers and industrial enterprises, but also focus on science and innovation (European Commission, 2011; Kleiner, 2011; Vey, 2010). However, in the case of the Spanish framework, it must be stressed that the Spanish National Evaluation and Foresight Agency (ANEP, Agencia Nacional de Evaluacion y Prospectiva) is independent from the funding institutions (Vey, 2010). The last framework, from the Netherlands, is a more specialized tool for evaluating societal impact of applied health research in order to complement the national system of quality assurance for all academic research (Royal Netherlands Academy of Arts and Sciences, 2002). In regard to the evaluation procedure of the referenced frameworks, three are primarily based on a peer review system, namely the Methodology for Evaluation of the Societal Impact of (applied) Health Research in the Netherlands, the Spanish National Evaluation and Foresight Agency, and German Excellence Initiative Strategy (Royal Netherlands Academy of Arts and Sciences, 2002; Kleiner, 2011; Vey, 2010). The remaining five frameworks apply a mixed-method evaluation procedure which, in addition to peer review, use bibliometrics or other type of quantifiable indicators, specifically the REF, Quality Assessment Mechanism of the United Kingdom RAENG, Horizon 2020, ARQF, Italian VTR (Higher Education Funding Council for England,

Scottish Funding Council, Higher Education Funding Council for Wales, & Department for Employment and Learning, 2009; The Royal Academy of Engineering, 2000; European Commission, 2011; Butler, 2008; Franceschet, & Costantini, 2011). Molas-Gallart, and Tang (2007) provide more quantifiable measurements for research impact evaluation such as willingness-to-pay or career tracking. Another two frameworks were created for evaluating research coming from only one source of research production. This is the case of the Quantitative Research Assessment Tool and Qualitative Research Assessment Tool. The first one evaluates merit of quantitative research articles published on the Child Care and Early Education Research Connections website and the second one evaluates accuracy of qualitative research articles published on the website (Child Care & Early Education Research Connections, n.d.). In contrast, five frameworks are intended to be flexible enough to use for multiple purposes or in differing research contexts, namely the Payback Framework, SIAMPI, Elsevier’s, SCImago’s, and Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0. The Payback Framework was originally developed for application by health research centers and uses logic models to represent the research processes and includes a series of categories to classify the individual impacts from research (Donovan & Hanney, 2011; Hanney, Gonzalez-Block, Buxton, Kogan, 2003). It has been applied in a social science research evaluation context with a modified set of categorized paybacks (impacts), specifically in the United Kingdom Economic and Social Research Council’s (ESRC) Future of Work program (Klautzer, Hanney, Nason, Rubin, Grant, & Wooding, 2011). Another adaptation of this framework was developed by Wooding, Hanney, Buxton, & Grant (2004) and was used to evaluate 16 research grants awarded by the Arthritis Research Campaign (arc) in the beginning of the 1990s. SIAMPI focuses on the activities of both researchers and other stakeholders for the identification of three types of ‘productive interactions’ (direct, indirect, and financial interactions) in creating social impacts (Spaapen & van Drooge, 2011). The data gathered are field-specific indicators that provide information about contribution and uptake from stakeholders. Thus, this method is more focused on the contribution of specific actors and the exchange of knowledge and expertise of different stakeholders, rather than attribution and impact (Spaapen & van Drooge, 2011). This framework has been applied in healthcare, information and communications technology, nano-sciences, and social sciences (Spaapen & van Drooge, 2011). Based on the availability of multi-disciplinary research, Elsevier’s Measure of Excellence and SCImago’s Evaluation Framework of Research Performance can be widely applied (SCImago Lab, 2011; Wellings & Winzer, 2011). The first approach assesses research publications and co-citations in order to identify the following: (1) number of publications (publication leadership), (2) number of citations (reference leadership), and (3) a measure of novelty (state of the art leadership). Every criterion is analyzed in comparison to global activity or competing countries. Finally, the process involves multi-disciplinary research topics (Wellings & Winzer, 2011). Similarly, the methodology and criteria of

evaluation of the second framework is based on six indicators: (1) output, (2) international collaboration, (3) normalized impact, (4) high quality publications, (5) specialization index, and (6) excellence rate. These indicators are intended to “…unveil some of the main dimensions of research performance of worldwide research-devoted institutions” (SCImago Lab, 2011, p. 1). Finally, the Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 can be used when assessing primary studies for inclusion in a meta-analysis. However, even a meta-analysis could be considered a standardized procedure applied “…to understand the results of any study in the context of all the other studies” (Borenstein, Hedges, Higgins, & Rothstein, 2009, p. 9).

Users of Existing Frameworks There are two types of users of the frameworks: active users (those who actually apply the framework) and passive users (those who use the results of the framework applied). The active group largely consists of research councils, funding councils, and governmental departments in the countries in which the frameworks originate. For instance, this is the case for the German Excellence Initiative Strategy, United Kingdom Research Excellence Framework (REF) for higher education institutions, Spanish National Evaluation and Foresight Agency, and Italian VTR (Kleiner, 2011; Butler, 2008; Vey, 2010; Franceschet, & Costantini, 2011; Reale, Barbara, & Costantini, 2007). In regard to ARQF, Quality Assessment Mechanism of the United Kingdom Royal Academy of Engineering, and Horizon 2020, it can be stressed that these frameworks have the same intended users when they are applied officially (Donovan, 2011; The Royal Academy of Engineering, 2000; European Commission, 2011). A special case is related to the Health-Consumer Powerhouse Indices because the framework’s users—healthcare providers—belong to the European Union member states and Canada (Tijssen, Visser, & van Leeuwen, 2002). For other frameworks such as the Payback Framework, SIAMPI, methodology for evaluation of the societal impact of (applied) health research, Elsevier’s Measure of Excellence, Qualitative Research Tool, Quantitative Research Tool, and Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0, universities, (specialized) research centers, scholars, and individuals represent the active group of users (Donovan, 2011; Spaapen & van Drooge, 2011; Royal Netherlands Academy of Arts and Sciences, 2002; Wellings & Winzer, 2011; Child Care & Early Education Research Connections, n.d.). There is only one framework whose active user is the institution that created it. It is SCImago’s Evaluation Framework of Research Performance and it is applied annually (since 2009) by SCImago Lab. Two frameworks are used more globally, namely the Payback Framework (the United Kingdom, Ireland, Canada, Australia, Germany) and SIAMPI (the Netherlands, France, Spain, and United Kingdom). In regard to the second group of users (passive), researchers are considered in this group because they are often interested in published papers that apply the

diverse frameworks in different contexts. Policy-makers, research managers, media, general public, and researchers also are interested in research performance of different institutions and for SCImago Rankings they are the intended target audience (SCImago Lab, 2011). Nevertheless, the results provided by the Spanish National Evaluation and Foresight Agency are utilized by the funding institution, which allocates resources for research and development.

Strengths of Existing Frameworks Several strengths emerge from the identified frameworks, including transparent processes; impacts supported by groups of indicators; the advance identification of stakeholders; improved uses of bibliometrics for measuring excellence; and controlling the management of research funding. Measuring research quality through peer review implies the development of a transparent process. The ARQF (Meek, & Van der Lee, 2005; Butler, 2008) and Italian VTR (Reale, Barbara, & Costantini, 2007) have in common this strength. According to Butler (2008), the Australian framework is transparent and reliable and avoids undue complexity. The Italian VTR stresses transparency in the involvement of external experts in establishing peer review panels, use of multiple methods to communicate and disseminate results, and continuous supervision and conflict resolution provided by the Committee for the Evaluation of Research (Reale, Barbara, & Costantini, 2007). Research impacts include the social, cultural, environmental, and economic returns that research produce for wider society (Butler, 2008, Donovan, 2011; Higher Education Funding Council for England, Scottish Funding Council, Higher Education Funding Council for Wales, & Department for Employment and Learning, 2009; Richardson, 2011). Thus, the methodology for evaluation of the societal impact of (applied) health research and Horizon 2020 divided indicators into groups that support the assessment of impacts (Royal Netherlands Academy of Arts and Sciences, 2002; European Commission, 2011). The framework from the Netherlands also considers that stakeholders can be consulted about the objectives of the evaluation (Royal Netherlands Academy of Arts and Sciences, 2002). The SIAMPI approach indicates that the identification and involvement of all stakeholders is the key to creating research impacts (Spaapen & van Drooge, 2011). Based on this premise, Spaapen & van Drooge (2011) claim another strength is the identification of researchers and stakeholders in advance. Hence, the Quality Assessment Mechanism from the United Kingdom RAENG, German Excellence Initiative Strategy, Horizon 2020, and methodology for evaluation of the societal impact of (applied) health research facilitate the formation of a link between different levels of stakeholders such as researchers (universities, graduate schools, research centers), industry, market opportunities, and final users (The Royal Academy of Engineering, 2000; Deutsche Forschungsmeinschaft (DFG), 2011; European Commission, 2011; Royal Netherlands Academy of Arts and Sciences, 2002). Particularly

between 2005 and 2011 the German Initiative has allocated 1.9 billion Euros by promoting top-level research pooling together universities and research centers (DFG, 2011; Kleiner, 2011). Elsevier’s Measure of Excellence and SCImago’s Evaluation Framework of Research Performance have in common an improved application of bibliometrics for measuring excellence by including competency analysis, international comparison, interdisciplinary research, and a large percentage of worldwide scientific output (Wellings & Winzer, 2011; SCImago Lab, 2011). Finally, the Spanish Research Funding System as it was aforementioned includes a monitoring tool of the allocated resources for research, development, and innovation. This tool is expected to monitor and evaluate administrative issues and technical and strategic aspects (Vey, 2010).

Weaknesses of Existing Frameworks A general limitation of several frameworks is that bibliometric procedures fail to recognize the complexity and multidimensional construct of research quality (Butler, 2008). Butler (2008) points out that the ARQF does not have a single quantitative measure, or a ‘basket’ of indicators, that provides an accurate and unambiguous result. Elsevier and SCImago’s frameworks have two limitations, which are the limited source of journal articles (only Scopus) and the absence of arts and humanities disciplines in the analysis (Wellings & Winzer, 2011; SCImago Lab, 2011). In regard to the last aspect, new scientific areas will be added to the framework’s ranking system: health sciences, life sciences, physical sciences, and social sciences and humanities (SCImago Lab, 2011). In terms of article quality, it is hardly assessed and there are citation time lags in disciplines (Wellings & Winzer, 2011). Another limitation is the short level of reliability of evidence on research excellence based on journal citations (Wellings & Winzer, 2011). In applying the Italian VTR framework, Van Raan (2005) and Reale, Barbara, & Costantini (2007) found a weak correlation between peer review scores and index factors, suggesting index factors are not a strong predictor for the quality of research manuscripts. Instead, index factors are a good predictor of journal quality (Van Raan, 2005; Reale, Barbara, & Costantini, 2007). In other words, “…impact factor is a good predictor of the quality of journals – not for the quality of articles published in a particular journal” (Franceschet & Costantini, 2011, pp. 15-16). Weaknesses include high implementation costs for participating institutions in the Research Excellence Framework (REF) (Bekhardia, 2009) and issues with the research excellence criteria in the REF and Italian VTR. While it is suggested that the REF should not apply the same criteria of research excellence to all disciplines (Bekhradnia, 2009); the VTR presents a lack of transparency about research excellence criteria used in peer reviews (Reale, Barbara, & Costantini, 2007).

Conclusions: Key Issues and Trends Although research excellence is not well defined in the literature, many of the existing frameworks for evaluating research excellence share certain features in terms of the dimensions or indicators of research excellence considered, their specific purposes, intended uses, intended users, strengths and limitations (see Table 1).

Table 1 Trends across Frameworks

Category Trends across Frameworks

Dimensions Research quality refers to epistemic, social, and economic robustness as well as importance of research and consistency of research results

Outputs encompassing volume, quality, impact, and utility Research impact as the social, cultural, environmental, and economic returns

that research outcomes produce for wider society, outside academia Research environment including research infrastructure and supporting activity

which contribute to dissemination and application of the research Purposes Allocation of public funds to universities and research centers’ research

Evaluation of research impacts in different contexts Making comparisons across different research organizations

Uses Context dependent, involving peer review or quantifiable indicators Topic-driven such as health research, social sciences, and hard sciences.

Users Vary from individual users to research councils, funding councils, and governmental departments

In some cases policy makers Strengths Transparent process

Impacts supported by groups of indicators Identification of researchers and stakeholders in advance Improved use of bibliometrics for measuring excellence though procedures

designed to recognize the complexity and multidimensional level of research excellence

Weaknesses Some frameworks have not been pilot tested High implementation costs Criteria for research excellence Conflicts between stakeholders and researchers Nonlinearity of the research process

References Bekhradnia, B. (2009). Proposals for the Research Excellence Framework – a

critique. Retrieved from http://www.hepi.ac.uk/files/43%20Proposals%20for%20the%20REF%20-%20a%20critique.pdf.

Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2009). Introduction to Meta‐Analysis. West Sussex, UK: Wiley.

Butler, L. (2008). Using a balanced approach to bibliometrics: quantitative performance measures in the Australian Research Quality Framework. Ethics in Science and Environmental Politics, 8, 83-92. doi: 10.3354/esep00077

Child Care & Early Education Research Connections (n.d.). Qualitative Research Assessment Tool. Retrieved from http://www.childcareresearch.org/childcare/datamethods/downloads/QualitativeResAssessTool.pdf

Child Care & Early Education Research Connections (n.d.). Quantitative Research Assessment Tool. Retrieved from http://www.childcareresearch.org/childcare/datamethods/downloads/quantitativeresearch.pdf

Deutsche Forschungsmeinschaft (DFG). (2011). Excellence Initiative at a glance. The programme by the German Federal and State Governments to promote top-level research at universities. Bon, Germany: Deutsche Forschungsmeinschaft (DFG). Retrieved from http://www.research-in-germany.de/main/research-landscape/r-d-policy-framework/28156/excellence-initiative.html

Donovan, C. (2008). The Australian Research Quality Framework: A live experiment in capturing the social, economic, environmental, and cultural returns of publicly funded Research. New Directions for Evaluation, 118, 47-60. doi: 10.1002/ev.260

Donovan, C. (2011). State of the art in assessing research impact: Introduction to a special issue. Research Evaluation, 20(3), 175-179. doi: 10.3152/095820211X13118583635918

Donovan, C. & Hanney, S. (2011). The ‘Payback Framework’ explained. Research Evaluation, 20(3), 181-183. doi: 10.3152/095820211X1311858363575

European Commission. (2011). Horizon 2020 - The Framework Programme for Research and Innovation. Brussels: European Commission. Retrieved from http://ec.europa.eu/research/horizon2020/pdf/proposals/horizon_2020_impact_assessment_report.pdf#view=fit&pagemode=none

Fournier, D. M. (1995). Establishing Evaluative Conclusions: A Distinction Between General and Working Logic. New Directions for Evaluation, 68, 15-32.

Franceschet, M., & Costantini, A. (2011). The first Italian research assessment exercise: A bibliometric perspective. Journal of Informetrics, 5(2), 275-291. doi: 10.1016/j.physletb.2003.10.071

Frontier Economics. (2009). Measuring the impact of ESRC funding: A Report prepared for the Economic and Social Research Council. London: Frontier Economics Ltd.

Furlong, J., & Oancea, A. (2005). Assessing quality in applied and practice-based educational research: A framework for discussion. Oxford, England: Oxford University Department of Educational Studies.

Geuna, A., & Martin, B. (2003). University research evaluation and funding: An international comparison. Minerva, 41(4), 277-304. doi: 10.1023/B:MINE.0000005155.70870.bd.

Hanney, C., Gonzalez-Block, M. A., Buxton, M. J., & Kogan, M. (2003). The utilisation of health research in policy-making: concepts, examples and methods of assessment. Health Research Policy and Systems, 1(2). doi: 10.1186/1478-4505-1-2

Higher Education Funding Council for England, Scottish Funding Council, Higher Education Funding Council for Wales, & Department for Employment and Learning. (2011). REF2014: Assessment framework and guidance on submissions. Bristol, England: Higher Education Funding Council for England. Retrieved from http://www.hefce.ac.uk/research/ref/pubs/2011/02_11/

Higher Education Funding Council for England, Scottish Funding Council, Higher Education Funding Council for Wales, & Department for Employment and Learning. (2009). Research excellence framework: Second consultation on the assessment and funding of research. Retrieved from http://www.ed.ac.uk/polopoly_fs/1.58331!fileManager/RKEC-PaperF-081009.pdf

Klautzer, L, Hanney, S., Nason, E., Rubin, J., Grant, J., & Wooding, S. (2011). Assessing policy and practice impacts of social science research: the application of the payback framework to assess the Future of Work programme. Research Evaluation, 20(3), 201-210. doi: 10.3152/095820211X13118583635675

Kleiner, M. (2011). The German Excellence Initiative. San Paulo: FAPESP Retrieved from http://www.fapesp.br/eventos/2011/08/initiative/Matthias_Kleiner.pdf

Meek, V. L., & Van der Lee, J. J. (2005). Performance indicators for assessing and benchmarking research capacities in universities. APEID, UNESCO Bangkok Occasional Paper Series. Retrieved from http://www2.unescobkk.org/elib/publications/05APEIDseries/APEIDseries_No2.pdf

Molas-Gallart, J., & Tang, P. (2007, March). Report of the ESRC Impact Evaluation Methods Workshop 20th March 2007. London: ESRC Workshop. Retrieved from http://www.esrc.ac.uk/_images/ESRC_Impact_Evaluation_Methods_Workshop_tcm8-3814.pdf

Molas-Gallart, J. (2012). Research Governance and the Role of Evaluation: A Comparative Study. American Journal of Evaluation, 0(0), 1-16. doi: 10.1177/1098214012450938

Nutley, S., Walter, I., & Davies, H. (2007). Using Evidence. How Research can Inform Public Services. Bristol: Policy Press.

Reale, E., Barbara, A., & Costantini, A. (2007). Peer review for the evaluation of academic research: Lessons from the Italian experience. Research Evaluation, 16(3), 216-228. doi: 10.3152/095820207X227501

Research Trends. (2008). The RAE: Measuring academic research. Research Trends, 3. Retrieved from http://www.researchtrends.com/category/issue3-january-2008/

Richardson, M. (2011). The value of bibliometrics. The Research Excellence Framework: Revisiting the RAE. Research Trends, 22. Retrieved from http://www.researchtrends.com/issue22-march-2011/the-research-excellence-framework-revisiting-the-rae-2/

Royal Netherlands Academy of Arts and Sciences. (2002). The societal impact of applied health research. Towards a quality assessment system. Amsterdam, Netherlands: Council for Medical Sciences.

SCImago Lab. (2011). SIR World Report 2011 - Global Ranking. Retrieved from www.SCImagoir.com/pdf/sir_2011_world_report.pdf

Spaapen, J., & van Drooge, L. (2011). Introducing ‘productive interactions’ in social impact assessment. Research Evaluation, 20(3): 211-218. doi: 10.3152/095820211X1294137187674

The Royal Academy of Engineering. (2000). Measuring excellence in engineering research. London: The Royal Academy of Engineering.

Tijssen, R. J. W., Visser, M. S., & van Leeuwen, T. N. (2002). Benchmarking international scientific excellence: Are highly cited research papers an appropriate frame of reference? Scientometrics, 54(3), 381-397.

Van Raan, A. F. J. (2005). Fatal attraction: Conceptual and methodological problems in the ranking of universities by bibliometric methods. Scientometrics, 62(1), 133-143.

Vey, L. (2010, June 25). The research funding system in Spain: Directorate General for Research and Management of the National R+D+I Plan presentation.

Wellings, P., & Winzer, R. (2011). Mapping research excellence: Exploring the links between research excellence and research funding policy. London: 1994 Group.

West, S., King, V., Carey, T.S., Lohr, K.N., McKoy, N., Sutton, S. F., & Lux, L. (2002, April). Systems to rate the strength of scientific evidence. Evidence Report/Technology Assessment No. 47 (Prepared by the Research Triangle Institute–University of North Carolina Evidence-based Practice Center under Contract No. 290-97-0011). AHRQ Publication No. 02-E016. Rockville, MD: Agency for Healthcare Research and Quality.

Wooding, S., Hanney, S., Buxton, M., & Grant, J. (2004). The returns from arthritis research. Volume 1: Approach, analysis and recommendations. Netherlands: RAND Europe.

Glossary of Abbreviations ANEP Agencia Nacional de Evaluacion y Prospectiva ARQF Australian Research Quality Framework ESRC United Kingdom Economic and Social Research Council DFG Deutsche Forschungsmeinschaft Horizon 2020 Common Strategic Framework for European Union Research

and Innovation Funding IDRC International Development Research Center Italian VTR Valutazione Triennale della Ricerca RAENG Quality Assessment Mechanism of the United Kingdom

Royal Academy of Engineering REF Research Excellence Framework