Upload
vuminh
View
214
Download
0
Embed Size (px)
Citation preview
Science,TechnologyandInnova2onindicators(STI2017).September6-8,2017-Paris,France
DanieleChecchi,IreneMazzo1aandSandroMomigliano
Na#onalAgencyfortheEvalua#onofUniversi#esandResearchIns#tu#ons(ANVUR)ROMA,Italy
www.anvur.it
Science,TechnologyandInnova2onindicators(STI2017).September6-8,2017-Paris,France
Aimofthestudyandresultsinanutshell
• Performance based funding are oEen subject to the cri2cismthat they may produce cumulaAve vicious cycles, since worseperforming ins2tu2ons receive lessmoney,making itmore andmoredifficulttocatchupbeLerperformingones.
• In thisworkweprovidefirst evidenceon this issue, comparingthe results achieved by Italian universi2es in the two na2onalevalua2onexercises,whichtookplaceinItalyin2012and2016.
• AEer making the two exercises comparable, we find that thedispersion in researchqualityof the ItalianuniversiAes falls inthe second exercise. Convergence is largely due to changes inrela2ve performance of researchers par2cipa2ng to bothexercisesandtothehiringdecisionsofuniversi2es.
2
Science,TechnologyandInnova2onindicators(STI2017).September6-8,2017-Paris,France
Context:asimplifiedoverviewofthetwoItalianresearchassessmentexercises(1)
3
Bothexercises:
- followedthesameapproachbasedonacombina2onofpeerreviewandbibliometricmethods.Theyreferred,respec2vely, totheyears2004-2010and2011-14.
- Evalua2onwascarriedoutbypanelsofexperts(about450expertsin eachexercise,with61whopar2cipated toboth)who recruitedabout15,000externalrefereesineachexercise
- were highly publicized, making a noteworthy impact on thereputa2onofins2tu2onsinvolved
- hadadirectimpactonthepublicfundingoftheuniversi2es,basedontheaverageperformanceoftheirresearchers’staff.Comparedtoapropor2onaldistribu2on,theschememodifiedalloca2onoftotalpublicfundstoindividualins2tu2onsbyupto4.5%(onaverageby2%)inthesecondexercise.
Science,TechnologyandInnova2onindicators(STI2017).September6-8,2017-Paris,France
Context:asimplifiedoverviewofthetwoItalianresearchassessmentexercises(2)
4
•
Science,TechnologyandInnova2onindicators(STI2017).September6-8,2017-Paris,France
Basicdata
5
QuarAlesVQR2004-2010 VQR2011-2014
#universiAes #researchers #universiAes #researchers
1stquarAle 22 35,415 22 33,188
2ndquarAle 23 14,075 23 13,540
3rdquarAle 23 4,645 23 4,753
4thquarAle 23 823 23 1,095
Total 91 54,958 91 52,576
Theuniversi2esthatpar2cipatedtotheexerciseshadverydifferentsizes,intermsofresearchersstaff.
Science,TechnologyandInnova2onindicators(STI2017).September6-8,2017-Paris,France
HarmonizaAonofgrades
6
ResearchproductallocaAon VQR2004-2010 VQR2011-2014 HarmonisaAon
Fraud (self)plagiarism:-2 notassessable:0
limited:0
Wrongdeliverables notassessable:-1 notassessable:0
Absenceofdeliverables missing:-0.5 notassessable:0
Decile*1
limited:0
limited:0Decile*2
Decile*3
acceptable:0.1Decile*4
Decile*5
Decile*6 acceptable:0.5fair:0.4 fair:0.4
Decile*7good:0.8
Decile*8high:0.7 high:0.7
Decile*9excellent:1
Decile*10 excellent:1 excellent:1
Ex-antemeanscore 0.41 0.35 0.32
Ex-antestd.dev.ofscores 0.43 0.33 0.36
*Decileintheworlddistribu2onofquality
Science,TechnologyandInnova2onindicators(STI2017).September6-8,2017-Paris,France
DistribuAonofuniversiAes’scoresinthetworesearchassessmentexercises(1)
Fortes2ngtheconvergence/divergenceofuniversi2esintermsofqualityofresearch,wecomputethefollowingsta2s2cs
7
Science,TechnologyandInnova2onindicators(STI2017).September6-8,2017-Paris,France8
DistribuAonofuniversiAes’scoresinthetworesearchassessmentexercises(2)
0,0
0,5
1,0
1,5
2,0
2,5
3,0
-1,0 -0,8 -0,6 -0,4 -0,2 0,0 0,2 0,4 0,6 0,8 1,0 1,2
VQR2011-2014H VQR2004-2010 VQR2004-2010HStd.dev. 0,190,27 0,25WeightedStd.dev. 0,11 0,16 0,14
• “VQR2011-14H”isvisiblymoreconcentratedaroundthemeanthanVQR2004-10.
• Theharmoniza2onreducesthedeclineinthedispersionofthescoresbetweenthetwoexercises.
• Convergencebetweenthetwoharmonizedexercisesissta2s2callysignificantwithFvalueequalto1.77(p-value<1%).
Science,TechnologyandInnova2onindicators(STI2017).September6-8,2017-Paris,France
AmodelforexaminingthedynamicsofuniversiAes’grades
9
whereγmeasuresthedependencefromini2alcondi2ons:• 𝛾<0impliesconvergence(i.e.regressiontothemean);thecloserthe<0impliesconvergence(i.e.regressiontothemean);thecloserthe
es2mated𝛾to-1thequickeristheconvergenceto-1thequickeristheconvergence• 𝛾>0(oritisbelow-2)impliesdivergence>0(oritisbelow-2)impliesdivergence
For beLer understanding the causes of convergence we split theresearchers involved in the two exercises (R1 and R2, respec2vely) intosubgroups,accordingtothefollowingdecomposi2on:
“perm”!whopar2cipatedinbothexerciseswiththesameacademicposi2on“1only”!whopar2cipatedonlytothefirstexercise“2only”!whopar2cipatedonlytothesecondorwerepromotedaEerthefirst
exercise.
Science,TechnologyandInnova2onindicators(STI2017).September6-8,2017-Paris,France10
Applyingthemodelto4differentpopulaAons(1)
• Highlysignificantnega2veregressioncoefficients
=>CONVERGENCE
• Coefficientofthe4thregressionisthelargest(inabsolutevalue)=>relaAvelylargecontribuAontoconvergenceofrecruitedandpromoted(R2only).
Science,TechnologyandInnova2onindicators(STI2017).September6-8,2017-Paris,France
R1+R2:theregressioncoefficient(-0.383)showsconvergence.Onaverage,inthesecondexerciseuniversiAesreducebymorethanathirdtheiriniAaldistancefromthemeangrading.
Rperm:theregressioncoefficient(-0,405)showsafasterconvergencetothemeanifweconsideronlythoseresearchersparAcipaAngtobothexerciseswiththesameacademicposi2on.
Rperm+R1only:includingtheeffectscomingfromreArement(largelydeterminedbyage),theregressioncoefficient(-0,372)getsclosertozero.
Rperm+R2only:theregressioncoefficient(-0.413)isthehighest(inabsolutevalue),sugges2ngarelaAvelylargecontribuAontotheconvergencecomingfromuniversiAesdecisionsonrecruitmentandpromoAons.Astheperformancerelatedschemeislargelytargetedtouniversi2es,itisherethatweshouldseethelargestimpactofthereputa2onandmonetaryincen2ves.
11
Applyingthemodelto4differentpopulaAons(2)
Science,TechnologyandInnova2onindicators(STI2017).September6-8,2017-Paris,France
Conclusionsandfuturework(1)
In this paper we look at the performance of individualItalianuniversi2es in the twona2onalevalua2onexercisesof research quality, which took place in Italy in 2012 and2016.
We find that, contrary to what was expected by cri2cs ofperformancebasedfunding,intheItaliancontexttherehasbeenasignificantconvergenceinthequalityofresearchofinvolvedinsAtuAons.
12
Science,TechnologyandInnova2onindicators(STI2017).September6-8,2017-Paris,France
Conclusionsandfuturework(2)
Our results suggest that this convergence was induced by themonetary and reputaAon incenAves. Apparently, the fact thatworse performing ins2tu2ons received less money did nothampertheireffortstocatchupbeLerperformingones.
Thesearepreliminaryresultsandshouldbetakenwithcau2on.Theycouldbetheeffectoftherela2velysmallamountoffundsshigedbythescheme;atmost4.5%ofpublicfundsreceivedbythe individual universi2es. Also, the convergence may be aniniAal(notpermanent)effectofthescheme.
Weplanto furtherstudythis issue,examiningthedeterminantsof the convergence and refining our harmoniza2on of the twoexercises.
13
Science,TechnologyandInnova2onindicators(STI2017).September6-8,2017-Paris,France
Thanksforyoura1enAon!
14