MAKING GREY LITERATURE COUNT

ERA, impact and other mechanisms for assessing and valuing grey

literature

Ian McMahon

Director, Research Group of Eight Ltd Canberra

Email: ian.mcmahon@go8.edu.au

Assessment of University Research Performance

• Promotion

• Internal university assessments

• Grant applications

• Excellence in Research For Australia (ERA)

• Assessment of Research Impact Trial

The Excellence in Research for Australia (ERA)

• aims to identify and promote excellence across the full spectrum of research activity

• evaluate university research

ERA Indicatorsresearch quality

•publishing behaviour, citation analysis, ERA peer review, and peer reviewed Australian and international research income.

research volume and activity

research application

esteem

ERA Eligible Research Output Types

Traditional research outputs

•Books—Authored Research; •Book—Chapters in Research Book; •Journal Articles—Refereed, Scholarly Journal; and•Conference Publications—Full Paper Refereed.

ERA ‘non-traditional’ research output

• Original Creative Works;

• Live Performance of Creative Works;

• Recorded/Rendered Creative Works; and

• Curated or Produced Substantial Public Exhibitions and Events.

• Visual art work

• Design/Architectural work

• Textual work novel or art review. Exhibition catalogues and catalogue entries

• Other: For example, scholarly editions, scholarly translations and public policy reports may be submitted under this category, provided they meet the relevant eligibility criteria, including meeting the definition of research.

Original Creative Works

For non-traditional research outputs which are nominated for ERA peer review, a statement identifying the research component of the output must be provided as part of the submission of an institution. The statement must be no more than 2000 characters (around 250 words)

Research Statement for ERA Peer Review of Non-Traditional

Research Outputs

ATN – Go8 Excellence in Innovation for Australia (EIA) Trial

• Trial of an assessment of the impact of research based on case studies.

• Jointly conducted by ATN and Go8 with the involvement of three other universities

• Recognise work in UK REF – pilot in 2010, full REF in 2014

• DIISRTE feasibility study into assessment of impact

Design Principles of the TrialThe EIA Trial aims to :

a) Identify good impacts of research

b) Justify the claims being made by institutions of the impacts

c) Identify the research underpinning the impact

d) Communicate to government, business and the community

e) Advisory Board including: Phil Clark, Robin Batterham, Patricia Kelly, Laurie Hammond, Craig Roy, David Sweeney + uni representatives

Guidelines for the Trial Four digit SEO; Four broad sectors, Defence,

Economic Development, Society, Environment Time period – impacts from 1st January 2007,

underpinning research from 1st January, 1992 Unit of evaluation – the impact itself, not

discipline, academic unit or individual A maximum of five case studies for each of the

four SEO Sectors (i.e. 20 in total) A minimum of two case studies in three of the four

SEO Sectors (i.e. 6 in total)

The mechanics of submissions Case Studies on prescribed templates ≤ 15

pages Information sought on

Context Details of the impact Underpinning research and research publications

– submissions to draw the link to research quality Income and other relevant metrics Validation material available People involved in research and impact

Evidence of Impact In making submissions universities need to

make provide evidence of impact Nature of evidence depends on nature of

impact Could include all forms of ‘grey literature’ Underpinning research could also have been

published in ‘grey literature’ Confidentiality issues in assessment

The assessment of submissions

• Each case study rated A-E based on reach & significance of impact

• 7 assessment panelso 1 x Defenceo 2 x Economic Developmento 2 x Societyo 2 x Environment

• Majority of panel members will be non-academics

Use of the results

A consolidated assessment report will be prepared for each institution on results for those units submitted from their own institution

A summary report of results spanning all institutions will be made available to each participating institution

A summary EIA Trial report will be prepared for the peak EIA Governance Group and released to the public following endorsement. This report will evaluate the process of the EIA Trial and make recommendations concerning its potential more widespread adoption

Higher Education Research Data Collection (HERDC)

• Collects research income and research output data for use by DIISRTE in funding research block grants

• Research outputs: refereed Journal Articles, Research Books, Book Chapters and Conference Papers

• Initial HERDC collections included other categories

• Internal university measures could include other categories eg policy reports

A Books

A1Authored- Research

A2Authored- Other

A3 Edited

A4Revision/New Edition

B Book Chapter

C Journal Articles

C1

Article in Scholarly Refereed Journal

C2

Other Contribution to Refereed Journal

C3Non-refereed Articles

C4 Letter or Note

D Major Reviews

E Conference Publications

E1 Full Written Paper- Refereed Proceedings

E2 Full Written Paper- Non-refereed Proceedings

E3 Extract of Paper

E4 Edited Volume of Conference Proceedings

F Audio-Visual Recordings

G Computer Software

H Technical Drawing/Architectural & Industrial Design/Working Model

I Patents

J Other Creative Works

J1 Major Written or Recorded Work

J2 Minor Written or Recorded Work

J3 Individual Exhibition of Original Art

J4 Representation of Original Art

Promotion criteriaThe following factors may be taken into account in assessing performance in research and/or creative activity:

•Publications

•IP

•evidence of professional activity in the visual and performing arts

•evidence of consultancy or professional work which contributes to advancement of knowledge, or scholarship within a discipline or professional field

•evidence of success in securing external funding for research

•evidence of a significant role in collaborative research projects

Promotion criteriaThe following factors may be taken into account in assessing performance in research and/or creative activity:•quality and impact of research and/or creative activity as evidenced by honours (prizes, higher degrees, professional qualifications, fellowships etc)•quality and impact of research and/or creative activity as evidenced by extent and nature of citations of work•quality and impact of research and/or creative activity as evidenced by successful uptake of intellectual property •quality and impact of creative activity as evidenced by representation in publications and collections and performance of works by other artists.

www.go8.edu.au

The Group of Eight LimitedPO Box 6229O’Connor ACT 2602

Level 2, Group of Eight House101 Northbourne AvenueTurner ACT 2612

Tel +61 2 6239 5844Fax +61 2 6239 5808executive.director@go8.edu.au

Esteem Measures

• Editorship of a Prestigious Work of Reference

• Fellowship of a Learned Academy and Membership of AIATSIS at 5.7.2.4

• Nationally Competitive Research Fellowships

• Membership of a Statutory Committee • Australia Council Grants or Australia Council

Fellowships

Indicators of research application

• research commercialisation income

• patents

• Plant Breeder‘s Rights

• registered designs

• NHMRC endorsed guidelines