Relevance and Impact of Humanities Research and how ERiC can help Jack Spaapen Royal Netherlands Academy of Arts and Sciences Humanities Conference Vienna

  • View

  • Download

Embed Size (px)

Text of Relevance and Impact of Humanities Research and how ERiC can help Jack Spaapen Royal Netherlands...

  • Slide 1

Relevance and Impact of Humanities Research and how ERiC can help Jack Spaapen Royal Netherlands Academy of Arts and Sciences Humanities Conference Vienna 15 and 16 December 2008 Slide 2 Relevance, impact and evaluation Various functions of evaluation Judgment about Quality, both scientific and societal Accountability to university, to government, to society Improvement of research quality and impact Various problems Functions get mixed in evaluations, but with different weight Some seem to be more obvious than others Criteria and indicators slanted towards scientific quality and to practices of natural and biomedical sciences Different actors have different expectations Peer review under pressure, evaluation procedures too Slide 3 Evaluating Research in Context ( endorsed by * Dutch Academy, * National research council (NWO), * Association of Dutch universities, * Council of the professional schools, * Rathenau Institute (Science Systems assessment) * QANU, * Ministry of Education and Sciences Slide 4 ERiC: Development of alternative evaluation methods WHY ERiC to inform boards / government about societal relevance of research (accountability and Lisbon ambitions) to help research areas that are not happy with dominant evaluation approaches : humanities, social sciences, technical sciences, health research, MIT, etc. Slide 5 Core problem The world is changing Science (academic) is changing into research (MIT) (Latour) Evaluation is not yet changing, or at least still dominated by criteria and indicators that used to work in the old situation Slide 6 methodological development RMW: the societal impact of health research [2002] SWR / RGW: judging research on its merits [2005] ERiC: evaluating research in context [2005, 2007] AWT: alfa stralen (humanities shine) disciplinary developments: health research (RGO / UMCs), development research (CERES); law (disciplinary committee) pay back, UK research councils, Scandinavian countries, etc. examples from outside Europe: USA, Australia, Canada Slide 7 Judging research on its merits [KNAW/SWR/RGW 2005] Collaboration of two Academy Councils, social sciences and humanities Dissatisfied with current evaluation systems mainly focusing on SCI (high impact journals) Two problems: 1. doesnt necessarily fit research communication patterns in soc sc and hum; 2. doesnt value societal output which is important for a lot of fields Search for more fitting evaluation schemes Slide 8 Necessary conditions for better evaluations Self evaluation focus on both scientific reputation and communication, and also on interaction with other audiences List of target groups: peers, students, professionals, policy makers, business, broader public List of indicators per target group: publications, citations, but also text books, reviews, grants from policy, collaborations with business, professionals, awards, popular publications, etc. Benchmarking is critical process Slide 9 Goals of ERiC stimulate debate about alternative methods of evaluation, fit for all disciplines regarding the value of research for society develop methods to do such evaluations website Slide 10 how does ERiC work? Context group as a central unit publications guidelines workshops, national and international pilot studies : agricultural research, pharmaceutical research, architecture, law, engineering European project : SIAMPI website : Slide 11 Results so far Working definition of Societal value: oAs a broad concept : societal relevance oAs a target-oriented concept : societal impact oAs a mainly economic concept : valorization these three have different consequences for evaluation Assessment approach fit for all disciplines: based on an analysis of the mission orientation of a research group oscientific community oprofessional sector oIndustry opolicy-oriented osociety at large Test method Slide 12 Test model: 4 steps 1.Reflection on mission orientation, self evaluation report 2.Empirical stage, find criteria and indicators for quality and relevance 3.Consultation with researchers and stakeholders, presentation of findings 4.Feed back and forward look Slide 13 Step 2 : productive interactions Identify social domains in context (policy, industry, society at large) Analyse the 4 main interaction channels between research and context: texts, people, artifacts, money Feed back to research community and stakeholders [workshops] Slide 14 example of evaluation of societal quality radar graph Slide 15 example REPP table graph Science, certified knowledge relative citation impact -- productivity scientific publications ++** (1 international visibility and collaborations = representation in editorial boards ++ invited lectures ++ Industry, market non-academic/commercial citing environment ++ productivity professional publications ++* (1 involvement in industry/market - advisory and expert roles in commercial domain -- editorships professional journal ++** Policy, societal involvement in policy domain + memberships and expert roles in governmental bodies ++ memberships of societal organisations: advisory/ education ++* production of public goods + additional grants from policy + Slide 16 example of evaluation of societal quality radar graph Slide 17 ERiC and the humanities Express the relations of humanities research with society in a meaningful way Help find criteria and indicators that might help to evaluate that Bring together the expertise of both stakeholders and researchers Stimulate them to find consensus about how to evaluate Slide 18 What can humanities do? dont try to emulate other fields try and find consensus as a field about indicators convince policy makers and managers that you are capable of defining indicators that are robust and fit for your field Slide 19 Netherlands France, NL, EU NL, UK Spain, UK Health Nanotechnology ICT Social sciences and humanities ERiC SIAMPI Slide 20 focus in human sciences 1.Describe the content, (societal) objectives, organization and evaluation practices of the program/projects involved in the pilot. 2.Identify research strategies and policies aiming at measuring the scientific and social impact of the program/projects in the various relevant dimensions. 3.Discuss the findings with the stakeholders related to this case, evaluating the validity, reliability and relevance of the results. 4.Advise on further development of methods for the assessment of social impact. Analysis of strength, weaknesses and possible improvements for the method to be developed. Slide 21 ERiC and SIAMPI and the Humanities 1.ERiC : cases in the Netherlands, but perhaps abroad 2.SIAMPI : dissemination of results through comments, conferences, joint workshops