Organizational complexity as a challenge to research assessment

Sally RumseyThe Bodleian LibrariesUniversity of Oxford

A case study of the University of Oxford

Complexity

A result of

• Size of the institution and the amount of research produced (research intensive)

Coupled with

• Organisational complexity of the institution (administrative structure; governance; etc)

University of Oxford Facts and figures*

• > 5800 academic and research staff• > 5600 research students• 4 university academic divisions (> 70 departments)

– Humanities: 9 faculties + Ruskin School of Art– Social Sciences: 16 departments– MPLS (Mathematical, Physical & Life Sciences): 12 departments– Medical Sciences: 16 departments– Plus cross disciplinary institutes eg Oxford Martin School and others

• 44 colleges and PPHs (Permanent Private Halls) (separate legal entities to University)

• External research funding £478.3m• Approx 4700 active research awards• Estimated 16,000 journal articles published p.a. • > 120 libraries* 2013 and 2013/14 figures: Annual Review 2013-14

Democracy

• Wellington Square as the centre of ‘the University’

• Democratic• Decision-making devolved to

the four Academic Divisions and their departments/faculties

• Decision-making can progress slowly

• Example: Symplectic adoption for REF 2014

• REF OA Service design project (for REF 2020) Cppyright Clare Byers

Colleges

• Separate legal entities to the university

• Dual appointments• College-only

appointed staff• Not on HR system• Submitted to REF

Copyright Jason Partridge

Governance and infrastructure

• Complex committee structure • Routes and timing of committees• Systems integration (WIP)• RIM data quality• Lack of unique identifiers and controlled vocabularies

across the entire institution (Divisions, Colleges, NHS etc)

• Devolved model • Internal matters also relevant to external integration• Compliance with standards (eg RIOXX)

Facts and figuresOxford REF 2014 submission

Submitted to 31 Units of Assessment

(UoA)* (out of 36)

c.2400 research staff

submitted

c. 8500 outputs

submitted

Returned 380,000 data

fields

REF 2008 resulted in £1.9 billion research income over 6

year period

Reported 4892.5 doctoral

degrees awarded

Took 2½ years to manage and compile return

* http://www.ref.ac.uk/panels/unitsofassessment/

REF complexity illustrative caseOII – Oxford Internet Institute

• A department of Social Sciences Division• Multi/cross disciplinary• Submitted under 9 UoAs (inc psychology,

economics, law, geography)• Difficulty in identifying which are the ‘best’

outputs. Best for whom?

Quality control for REF Case studies

• Quality control was a challenge• 31 UoAs split across 4 academic divisions• Needed consistency across all submitted case

studies across:– Unit– Within the division– Across all 4 divisions

REF complexity illustrative case: Impact

• Medical sciences – broad areas such as ‘cancer’ or ‘neuroscience’ over a large academic division

• Do not ‘sit’ in any one department• Administrative staff cannot know all the

details to report impact• Impact can be reported back 20 years

previously. Difficult to identify and track back so far – people leave. Generally cite only 10 years

The effect on assessment

• Locating data required for assessment

• Dealing with a variety of systems

• Lack of single location for each data type

• Data entry/import (quantity)• Consistency• Size of the operation and

numbers of people involved

Copyright Clare Byers

• REF OA Service Design Project

• Pilot Jan – June 2015• Simple, short message

– For individuals– For support staff

(including directive)• Comms is key• RS/Bod/Divisional staff

ORCIDs

• How ORCIDs should help– De facto standard– Common across all units of the University– Supports integration externally– To be required for REF 2020?– Ultimately link individuals, publications, funding– Assist automation

• Difficulties eg College staff not on HR/CUD system• Limitations• ORCIDs at Oxford service

ORCIDs at Oxford service is live

Released 12 May 2015

Points for discussion

• Data quality• Systems integration• Standards compliance• Common framework

between agencies• Reduce

administrative burden

• Automation to cope with scale Sally Rumsey, the Bodleian Libraries, 2015