Upload
buikien
View
217
Download
0
Embed Size (px)
Citation preview
Benchmarking and Appropriate use of Metrics to support Research Administration
Colin CooperResearch Facilitator
Liverpool Hope University
Contains a bit of controversy.
Benchmarking and Appropriate use of Metrics to support Research Administration
Content:• Benchmarking• Metrics • Use of Metrics• Other Sectors• Performance Metrics• New Metrics for RSO
Benchmarking
Definition
“Benchmarking is a tool for improving performance by learning from best practices and the processes by which it has been achieved.
Benchmarking involves looking how others achieve their performance levels. In this way benchmarking helps explain the processes behind excellent performance.”
Metrics
Definition
Metrics measure and report an organisation's performance and capture evidence based figures. It allows and encourages comparison and supports a strategy.
Statistics
Definition
The science of producing unreliable facts from reliable figures.
If metrics are evidence based figures, statistics are how we then elaborate (lie) about them!
Benchmarking
End Results• A set of meaningful performance information.• Improve strategic planning and provide an assessment
of strengths and weaknesses.• Ability to set challenging performance goals and
stimulate management of the research portfolio• Encourage implementation of best practices.• Lead to increased efficiency in the use of resources both
financial and physical.• Close the performance gap• Empowering of staff to seek further improvements• Get additional resources
BenchmarkingPerformanceBenchmarking
Comparison of performance measures to decide how good University is compared to others
Process Benchmarking
Processes are compared to others to improve your own
StrategicBenchmarking
Comparison to change strategic direction
Internal Benchmarking
Comparison made between Departments / School / Faculty in a devolved administration
CompetitiveBenchmarking
Comparison against like for like competition
FunctionalBenchmarking
Comparison of enterprise systems e.g. Finance, HR, Students: to become the best in use of current systems
Generic Benchmarking
Comparison against best in the business regardless of sector e.g. Amazon for marketing and Tesco CRM
BenchmarkingInstitutional Benchmarking • What do you want information for ?
To set goalsTo chose collaboratorsTo gee up your academic staff
• What / who do you measure ?Research income / Funding baseSuccess RatesPublications, patents, licences, ImpactNumber of Students @ UG, PGR,PGTStaff numbers, Research Staff numbersComparator / aspirational / Individuals/ Groups/ Departments / Faculties/ research centres
• Where do you get the information ?Research Excellence Framework (UK)HESA ( Higher Education Statistics Agency (UK)Funders StatsColleagues League Tables
Benchmarking STEP 1Sit back and think !“Not everything that can be measured is important and
not everything that is important can be measured” Albert Einstein
• Who’s asked for it ? Top down or bottom up• Costs (carrying it out / implementing results ) • Either way get yourself a “champion”• Team selection• What’s being benchmarked (priorities v KPI’s)• Select non friendly academic staff• Understand your own processes • Carry out your own data collection• Performance metrics
Benchmarking STEP 1Go deeper basic functions (Process
Benchmarking)
• Length of time to prepare costing • Length of time to negotiate a contract• Length of time from award to account set up• Staff appointments / in period changes• Close out within x number of days• Timeliness of invoicing • Outstanding debtors over x days• Ethics approval• General response time• FEC recovery achieved
Benchmarking STEP 2Identify Best Practice Select Partners
(Performance, Generic & Internal Benchmarking)
• Select appropriate comparators internal
or external
• Within HE Sector• Outside HE• International• Ethics even in benchmarking• Age of data HESA or wait for REF results
Benchmarking Step 3Decide on Approach
How Soon You Need Results Benchmarking Alternatives
Within a weekReading about research adminSurfing the webTelephone interviews
One to two weeksExperienced ResearchAdministratorHire a consultant
Three to six weeks E-mail Questionnairesite visit
Two or more months Larger benchmarking / review
Benchmarking STEP 4
Implement Findings
• Correct team selection back at step 1 • One size does not fit all• Don’t just imitate……..innovate (Imperial, City, UMIST,
Manchester, Liverpool, Liverpool Hope)• Put in place quick wins• Communicate the results to encourage use• Decide if knock-on for other areas need addressing• Feedback to participants• Link data collection and management for the Benchmark
results to ongoing work
Benchmarking STEP 5
Assess & Review
• Give it some time• How was it for you• If objectives not met, what can you do• Did we use the right metrics• Can we see marked improvement• Make sure staff are actually implementing the changes• Feedback the results to your participants
Benchmarking STEP 6
START AGAIN ?
• Benchmarking should not be a one-off exercise• Aim for continuous improvement• Select new areas to be benchmarked• Changes to administrative structures• Changes to funding • Did you learn anything or do you still believe you are the
best
Benchmarking
Definition
“Benchmarking is a tool for improving performances by learning from best practices and the processes by which it has been achieved.
Benchmarking involves looking outward to examine how others achieve their performance levels. In this way benchmarking helps explain the processes behind excellent performance.”
But what about trying something like this with Metrics, lets be innovative, different and controversial ………..
Biomolecular ScienceApplications 9,033 69 - 398 311 3,223 - ##### 65 Aw ards 2,048 27 164 - 293 643 - 3,175 Chemical EngineeringApplications 4,265 450 24 103 548 - - 5,390 26 Aw ards 1,711 183 334 140 48 - 19 2,435 ChemistryApplications 4,487 138 - 41 - 326 - 4,992 22 Aw ards 2,581 - 12 20 224 71 18 2,926 DIASApplications 3,581 1,188 - - 295 - - 5,064 20 Aw ards 871 48 7 38 31 7 - 1,002 OptometryApplications 229 206 22 308 37 713 - 1,515 15 Aw ards 525 6 159 215 - 155 - 1,060
Reporting Aug Sep Oct Nov Dec Jan Feb Mar Apr May Jun JulNo of Visits 2 2 0 0 1 0 1 1 1Formal Presentations 0 0 0 0 0 0 0 0 0 1Signif icant Proposals 0 0 0 0 0 0 0 0 0Partnerships 0 0 0 0 0 0 0 0 0Other 0 0 0 0 0 0 0 0 0
Other ActivityFuels for the Future ws wsBBSRC BioinformaticsSOCI m mBioNow m nwEPICC mNW Chem InitiativeShell mCIBA mEPSRC Mock Panels wsFP5 Soybean mWilton / Fraunhofer m m m mBiotech Group mBioindustry TT wsUniv Manchester mNHS IP cMedical House m
UK Ch
arities
Other
Total V
alue
Total N
umber
Resea
rch Co
uncils
Departments Gover
nment
UK Ind
ustry
Overs
eas
Europe
an Co
mmiss
ion
Information Capture for Academic Metrics
h-index, Bibliometrics, Publications and Citations, Scholarly Output, Citation Count, Citations per Output, Field-Weighted Citation Impact, Outputs in Top Percentiles, i10index, Publications in Top Journal Percentiles, Collaboration Metrics, Collaboration Impact, Academic-Corporate Collaboration, Academic-Corporate Collaboration Impact, Esteem and Socio-Economic Impact Metrics, Public Engagement, Enterprise Activities / Economic Development Metrics, Intellectual Property Volume, Intellectual Property Income, Sustainable Spin-Offs, Spin-Off-Related Finances, Research Gate, Research Fish, Web of Knowledge
Altmetrics
Tweet citations, Facebook post citations, Blog citations, General web citationsGrey literature citations, Online syllabus citations, Online presentation citationsDiscussion forum citations, Mainstream media citations, Mendeley, CiteULike or Zotero bookmark counts. Library holdings (of books), Citations from Google Books, Linkedin
Metrics
Metrics
Available Hardware / Software etc
Current Research Information System (CRIS), Scopus, Web of Science, Google Scholar, Elsevier’s Pure, Digital Science’s Symplectic1, Thomson Reuters’ Converis, Star Metrics, Research in ViewSM, ePrints, dSpace. Snowball Metrics Exchange, SCIMAGO, Common European Research Information Format (CERIF), HESA, RCUK, THES tables, NSS, US Office of Management and Budget, PlumX, Impactstory, Research Gate, Microsoft Academic Search, Socialcite, Papercritic, Readmeter, Crowdometer, ORCID, Excellence in Research (Aus)
Out of hand………When a commissioned report by HEFCE recommends an award for an annual “Bad Metric” prize to the most egregious example of an inappropriate use of quantitative indicators in research management. (Metric Tide)
ESRC : Centre for Doctoral Training
The CDT must consist of REF UoA which fulfil all the following criteria :
• a greater than or equal to 50 per cent REF output (3*+4*)
• a greater than or equal to 50 per cent REF environment (3*+4*)
• a greater than or equal to than 50 per cent REF impact (3*+4*)
• a research volume equivalent to a minimum of five FTE staff with output at 3* or 4* (calculated by number of FTE staff submitted to REF2014 ‘multiplied by’ percentage of REF output at 3* or 4*).
Funding fixes
% of the submission meeting the standard for:
4* 3* 2* 1*
Overall 15 38 34 13
Outputs 11.4 48.6 28.6 11.4
Impact 40.0 0.0 30.0 30.0
Environment 0.0 37.5 62.5 0.0
Liverpool Hope REF scores in UoA 25 Education
Catch 22
Metrics (Stern Review UK)• “no metric can currently provide a like-for-like
replacement for REF peer review.”
• “a move to metrics carries risks for the reputation of REF and the quality assessments it makes.”
• “we have concluded that metrics should not replace peer review as the primary approach to the assessment in the next REF”.
• “widespread recognition that robust metrics for impact don’t yet exist, such that narrative case studies, assessed by peer review, remain the best option”.
Metrics in the REF
• We know who’s going to get what (within 10%) funding, even before the exercise is carried out. In the last exercise, 20 out of 130 (English) universities received 67% of the funding
• Can be no doubt whatsoever regarding concentration of research funding in the UK
• Exercise costs the sector £246M = 4% of Research Spend
• Occasionally will throw up a surprise, so that the funding algorithm needs changing
Metrics
OTHER SECTORS Industry
• By calculating revenue less the real costs of the products the customer is ordering and less the real cost of servicing the customer we are left with the customer contribution. The customer contribution is the figure to compare one customer to another
Positive contributors
Negative contributors
Most profitable
100% contribution
Productsor
Customers
Positive contributors
Negative contributors
A BakeryHalf the customers added no profit. Rationalising product mix and changing service
levels to specific customers added £1 million to the bottom
line within six months
Cumulative net margin
Least profitable
A Wholesale CompanyLoss-making customers typically
had low order values, more returns and manual systems. Many were moved to profitability by changing these characteristics. Others were
‘let go’ to competitors.
An Electricity Utility15% of 3 million customers were
unprofitable. Profiling their characteristics enabled the
company accurately to predict costly behaviours and avoid them – which delivered £8
million additional profit.
A Retail ChainProduct categories with
negative net margins accounted for 17% of the
company’s overall net margin. Careful analysis of the causes
of excessive gross margin erosion produced positive net
margins for nearly all categories averaging £0.5 million a year in each store
MetricsPerformance Metrics (Sponsor Contribution)
• Strategic fit• Time for proposal preparation• Success Rates• Cost Recovery• Average value of awards• Late payments • Large administrative burden• Repeat Business• Royalty payments• Restrictive contractual terms
(improve funding base)
Metrics
Performance Metrics ( Academic Staff)
• Number of proposals per FTE• What % staff bring in % research funding• % working as PI• £’s per FTE• Personal portfolio growth• Consultancy v Research income• Outputs (impact on scientific area / outside of academe)• Licence income per FTE• PGR per FTESETTING TARGETS
(Increased Competitiveness)
MetricsPerformance Metrics (Academic Contribution)
• Unresponsive• Time for proposal preparation• Success Rates• Cost Recovery• Average value of awards• Late Technical Reports• Always on the phone over nothing• One project at a time• No Outputs• Not research but consultancy• Own slush funds
(improve internal base)
A universal law: The Hook CurveCumulative contribution
Funder & Academic
100%
Con
trib
utio
n
Internal and External Stakeholders
Positive contributors
Negative contributors
100%
30% 60%Thanks to Brian Plowman
MetricsPerformance metrics (Admin)
• Number of proposals per FTE• Value of awards per FTE• Number of awards per FTE• Increase in funding over a fixed period• Cost of RSO as a % of £’s awarded
Pre & Post award• Number of PI’s per RSO FTE• Central or Devolved Administration• Go deeper basic functions
MetricsPerformance Metrics (Own Staff Contribution)
“We all dream of a team of Carraghers”
• Managing self• Managing people• Managing work
How can all that be achieved ?Individual Training plansTeam Training PlansRobust Appraisal System (not just Head of Office meetings)Team Spirit / inclusive office / Snr Academic staff participation
Using Benchmark and Metrics Findings in a FEC Environment
Core
Adds value, meets business and customer needs
Support
Enables core activityto take place
Diversionary
Adds cost, but no value -caused by process failure
Reduce costs:By changing the method
Reduce costs:By making an upstream process improvement
..and byChanging Funder and Academics behaviours
Reduce (increase) costs:
By changing the level of service
internally & externally and identify risks
(benefits)
Research proposals
Support of Funders
and Academics
Explaining to Funders why last project was
late, report had missing sections and the invoice
was wrongThanks to Brian Plowman
Becoming Competitive
51%30%
19%
Typical starting
point
Overall less and
rebalancedBecomes
Thanks to Brian Plowman
Win Win for all
New MetricsPerformance Metrics (Research Support Office)• Reducing bureaucracy
• Removing administration burden
• Systems and processes that work for all
• Impact of the office
• Influence on Stakeholder opinion
• Influence on Stakeholder awareness
• Peer recognition
• Improved happiness metrics
New MetricsUseful Metrics for our Academics / Faculty
• Distance to the Library
• Success rates for various schemes
• Value of funded applications by Sponsor
• Look at travel vs equipment vs staff costs
• Publishers who, when, how, current topics• https://sites.insead.edu/library/rankings/journal_rankings.cfm
• Technical reports frequency
New Metrics
Useful Metrics for our Academics / Faculty
• Journal impact factors (IF) for publications
• Collaborator metrics / Screening of potential partners
• Demand Management by Funders / University
• Teaching loads / administrative loads
• Impact routes and data for their research activity
• Grade Point Averages
• Personal Portfolio values
Basic Calculations for a GPA:Points Total = Sum of: Star-value X Number of articles at this Star-value level
e.g. for researcher in Table below:
Points Total = 3 X 3 + 2 X 1 = 11 (because there are 3 articles at star-value 3 and 1 at star-value 2)
GPA= Points Total/Outputs Required
For example in Table below:- GPA = 11/4 = 2.75 (because Points Total is 11 and Outputs required from this researcher is 4.
FTE Outputs Required
4* 3* 2* 1* GPA Points Total
1.00 4.00 3 1 2.75 11
Basic calculations:GPA= Points Total/Outputs Required
FTE Outputs Required
4* 3* 2* 1* GPA Points Total
0.60 2.00 2 3.00 61.00 4.00 3 1 2.75 111.00 4.00 1 3 3.25 131.00 4.00 1 3 3.25 131.00 4.00 1 3 3.25 131.00 4.00 1 3 3.25 131.00 4.00 2 2 3.50 141.00 4.00 1 2 1 3.00 12
Metrics
Colin’s Top Five Metrics
1. No proposals = No Awards
2. ? % staff bring in ? % research funding
3. ? % Sponsors supply what ? % research funding
4. What’s the happiness metric in your office
5. No metric can replace a gut feeling or local knowledge
Metrics
TEF Teaching Excellence Framework
• Metric Based
• Opt out ?
• National Student Survey
• What’s the benefit
• Do league tables count