View
0
Download
0
Category
Preview:
Citation preview
1
Whatcountsinresearch?Dysfunctioninknowledgecreation&movingbeyondbyHeatherMorrisonBiostatement:HeatherMorrisonisanAssociateProfessorintheSchoolofInformationStudies,andcross-appointedtotheDepartmentofCommunication,UniversityofOttawa.Herresearchfocusistheeconomicsoftransitionofscholarlypublishingfrompurchaseandsubscriptionstosupportforproductiontofacilitateopenaccess.HeatheristhePrincipleInvestigatoroftheSSHRCInsightGrant(2016-2021)supportedSustainingtheKnowledgeCommonsresearchprogram.AbstractThischapterbeginswithabriefhistoryofscholarlyjournalsandtheoriginsofbibliometricsandanoverviewofhowmetricsfeedintouniversityrankings.Journalimpactfactor(IF),ameasureofaveragecitationstoarticlesinaparticularjournal,wasthesoleuniversalstandardforassessingqualityofjournalsandarticlesuntilquiterecently.IFhasbeenwidelycritiqued;evenClarivateAnalytics,thepublisheroftheJournalCitationReports/IF,cautionsagainstuseofIFforresearchassessment.Inthepastfewyearstherehavebeenseveralmajorcallsforchangeinresearchassessment:the2012SanFranciscoDeclarationonResearchAssessment(DORA),the2015LeidenManifesto(translatedinto18languages)andthe2017ScienceEuropeNewvisionformeaningfulresearchassessment.Meanwhile,duetorapidchangeintheunderlyingtechnology,practiceischangingfarmorerapidlythanmostofusrealize.IFhasalreadylargelybeenreplacedbyitem-levelcitationdatafromElsevier’sScopusinuniversityrankings.Altmetricsillustratingawiderangeofusesincludingbutmovingbeyondcitationdata,suchasdownloadsandsocialmediauseareprominentlydisplayedonpublishers’websites.Thepurposeofthischapteristoprovideanoverviewofhowthesemetricsworkatpresent,tomovebeyondtechnicalcritique(reliabilityandvalidityofmetrics)tointroducemajorflawsinthelogicbehindmetrics-basedassessmentofresearch,andtocallforevenmoreradicalthoughtandchangetowardsamorequalitativeapproachtoassessment.ThecollectiveagreementoftheUniversityofOttawaispresentedasonemodelforchange.Abriefhistoryofjournals,bibliometrics&rankingsIn1665,twoscholarlyentrepreneursindependentlyseizedthepotentialoftheprintingpressandthepostalsystem,andinventedthemodernscholarlyjournal.Guédon’s(2001)InOldenburg’slongshadowpresentsanoverviewofthehistoryofthescholarlypeer-reviewedjournalfromitsinceptionin1665withOldenburg’sPhilosophicalTransactionsanddeSallo’sJournaldesSçavans,totheendofthe20thcentury.Theideaofpeerreviewhasevolvedovertime,buttheformatofjournalshasremainedlargelythesame.Odlyzko(1994)predictedtheimpendingdemiseofscholarlyjournals.Printandmailareintheprocessofbecomingobsoleteasthestandardforproductionanddisseminationofscholarlywork,asthisbecomeselectronicandweb-based.Thecontinuityoftheprint-basedformat,withonlinejournalscloselyresemblingprintones,reflectsacceptanceofthescholarlyjournalarticleasthegoldstandardforpublicationinmanyacademicdisciplines.Thegrowthofscholarlyjournalsandarticlessince1665hasbeenremarkablyconstant.ThiswasfirstdocumentedbyDeSollaPrice(1963,p.17)inLittleScience,BigScience,andupdatedbyMabe&Amin(2001)andMabe(2003).Thereisanaverageannualscholarly
2
journalandarticlegrowthrateofabout3-3.5%peryearfromthe1600stothepresentday.Iftherewerestilljusttwoscholarlyjournalsproducingasmallvolumeofarticlesonanannualbasis,itwouldbefeasibleforeveryscholartoreadeveryscholarlyarticle.However,asthevolumeofproductiongrew,journalsbegantospecializeinparticulardisciplinesandsub-disciplines,atratesvaryingwiththegrowthofthedisciplines.Asproductioncontinuedtoincrease,specializationwasnotenough.Guédon(2001)arguesthatthegrowingnumbersofjournalswastheinspirationforatendencytowanttodefine“corejournals”.Thepurposeofthe“corejournal”conceptwastoaddresstwoproblemsthataroseasthenumberofjournalsgrew.Oneproblemwasthe“serialscrisis”documentedbytheAssociationofResearchLibraries(1989),acombinationofincreasingnumbersofjournalsandaveragepricerisesforjournalsbeyondinflation,yearafteryear,leadinglibrariestocancelsubscriptions.Asecondproblemwastheincreasingdifficultyscholarshadinkeepingupwiththegrowingliterature.Inthatsense,identifying“corejournals”wouldhelpbusyscholarsprioritizetheirreadingsandpublicationvenues.Garfield(1955)proposed“abibliographicsystemforscienceliteraturethatcaneliminatetheuncriticalcitationoffraudulent,incomplete,orobsoletedatabymakingitpossiblefortheconscientiousscholartobeawareofcriticismsofearlierpapers”.Anotherproposedpurposeofthissystemwastofacilitatecommunicationamongscientists.ItwasinthisarticlethatGarfieldfirstcoinedtheterm“impactfactor”(IF),athen-hypotheticalmeasureoftheinfluenceofahighlycitedarticle.Garfield(2006)describesthehistoryofthedevelopmentofcitationindexingandIF.WithsupportfromtheU.S.NationalInstitutesofHealth,IFbecamethebasisforthedevelopmentoffirsttheGeneticsCitationIndexandlatertheScienceCitationIndex.IFisametricappliedtojournalsratherthanarticles.Itisbasedon2elements,anumeratorconsistingofthenumberofcitationsinthecurrentyeartoitemspublishedinaparticularjournalintheprevious2years,andthedenominator,thenumberofsubstantivereviewsandarticlespublishedinthesame2years.Inotherwords,IFistheaveragenumberofcitationstoanarticleinaparticularjournalfortheprevioustwoyears.IFvariesconsiderablybydisciplineandsub-disciplineaswellasbyjournal,andisoftenevaluatedonthebasisofthestatusbyquartilewithinadiscipline.TheScienceCitationIndexdevelopedbyGarfieldandcolleaguesin1961hasmorphedandgrownintoWebofScience,includingtheScienceCitationIndex,theSocialScienceCitationIndex,theArts&HumanitiesCitationIndex,theEmergingSourcesCitationIndex,theBookCitationIndex,theConferenceProceedingsCitationIndex,inadditiontooptionalspecializedcollections.WebofScienceisoneofasuiteofinter-relatedproductsproducedandsoldbyClarivateAnalytics(untilrecentlypublishedbyThomsonReuters),andthebasisfortheirJournalCitationReports(JCR)whichprovidesreportsofjournalIF.
3
Table1:coreClarivateproductsTable1illustratestherelationshipandevolutionofthecoreproductsrelatedtoresearchandresearchmetricsofferedbyClarivateanalytics.Theinitialcoreunderlyingproductisamassivedatabaseofcitationstojournalarticles,WebofScience.TheresearchdiscoverytoolcalledWebofScienceisextensivelyusedbyresearchersatuniversityandresearchlibrariesforresearchdiscovery.ThesameunderlyingmetadataisusedforthetraditionalJournalCitationReports(JCR).ThisisthetooloriginallyenvisionedbyGarfieldasameansofidentifyingasetof“core”ormosthighlycitedjournalssothatresearcherscouldprioritizetheseforreadingandlibrariesforpurchase.JCRisimportantasabrandingtoolforpublishers.JCRisalsousedbyuniversityrankingsagencies.Ontheright,morerecenttoolsInCitesandEssentialScienceIndicatorsfocusexclusivelyonmetricsforevaluation.Thesearetoolsformeasuringresearchers,notassistingresearchersintheirwork.Atasurfacelevel,informationondifferentpagesontheClarivatewebsitemightappearconfusingandcontradictory.ThisisbecauseClarivateoffersservicestoadiversegroupofstakeholdersthathavedifferentgoalsthatarenotalwayscompatible.OntheJCRwebsite,ClarivateclaimsthatJCR“givesyouasystematic,objectivemeanstoevaluatetheworld’sleadingscientificandscholarlyjournals.Byanalysingcitationreferences…JCRmeasuresresearchinfluenceandimpactatthejournalandcategorylevels,andshowstherelationshipbetweencitingandcitedjournals”(ClarivateAnalytics,n.d.).JCRincludes11,000journalsfromover230disciplines.Thisisaboutathirdofthetotalactivepeer-reviewedjournalsreportedbyWare&Mabe(2015)inarecentstateoftheindustryoverviewreportproducedfortheInternationalAssociationofScientific,TechnicalandMedicalPublishers(STM)(28,100intheEnglish-language,6,450inotherlanguagesthanEnglish).ThetargetmarketsforJCR,accordingtotheClarivatewebsiteasofOctober16,2018arelibrarians,toinformpurchaseandcancellationdecisions;publishersandeditors,toassesstheeffectivenessofjournalsinthemarketplace;researchers,toidentifythemostinfluentialjournalsinwhichtopublish,andresearchmanagersandinformationanalyststo“track
WebofScience(Citationmetadata)
ResearchDiscovery
WebofScience:universityandresearch
libraries
ResearchEvaluation
InCitesResearchorganizations,
funders
JounalCitationReports:universityandresearchlibraries,publishers,universityrankings
agencies
EssentialScienceIndicators
Researchorganizations,fundingagencies,
publishers
4
publicationandcitationpatternstoaidyourstrategyandpolicydecisions”.ClarivateAnalytic’s(n.d.)webadviceonthesuitabilityoftheuseofJCRandotherdataderivedfromWebofScienceinresearchassessmentisdependentonthetargetaudienceoftheirdifferentproducts.ThereisamarkedcontrastbetweenadviceontheJCRwebsite(librariansastheprimarytargetaudience)andInCites(researchorganizationsandfundingandpolicyorganizationsasthetwoprimarytargetaudiences).OntheJCRwebsite,ClarivatewarnsagainsttheuseofIFinassessingjournalquality,statingthat“ClarivateAnalyticsdoesnotdependontheimpactfactoraloneinassessingtheusefulnessofajournal,andneithershouldanyoneelse…Theimpactfactorshouldbeusedwithinformedpeerreview.Inthecaseofacademicevaluationfortenureitissometimesinappropriatetousetheimpactofthesourcejournaltoestimatetheexpectedfrequencyofarecentlypublishedarticle.Again,theimpactfactorshouldbeusedwithinformedpeerreview.Citationfrequenciesforindividualarticlesarequitevaried.”Incontrast,thetitleoftheClarivatewebsiteforInCites(likeJCR,basedonWebofSciencedata)statesthatitis“anobjectiveanalysisofpeople,programsandpeers[emphasisadded]”.RecommendeduseofInCitesdataforassessingindividualresearchersasaprimaryuseisimpliedforeachtargetaudienceintheir“Who’sitfor”.Forresearchorganizations,InCitesispresentedasameansto“identifyandmanageresearchactivitiesandtheirimpact”aswellasto“identifyexperts”.Forfundingorganizations,usessuggestedinclude“identifyemerging…researchersandexperts”and“managefundingactivityfromsubmissionstoprogressreportsthroughoutcomes”.ThisassumesthatemergingresearchersandexpertscanbeobjectivelyidentifiedthroughWebofSciencedata,i.e.researcherswhopublishinhighIFjournalsarepresumably“emerging”and“expert”.ItisalsoassumedthatpublicationinhighIFjournalsandhighcitationcountsareobjectivemeasuresofthequalityofresearch.PublishersaretoldthatInCitesprovidesameansto“identifythebestauthorsandreviewers”.WareandMabe(2015)discussincreasingindustrycriticismoftheuseofcitationdata,particularlyIF,tojudgethequalityofindividualresearchersanddepartments.Willthisindustryrecognitionleadtochange,andifsowhatformwillthischangetake?Basedonmajoruniversityrankingagencies’descriptionsoftheirmethodsandElsevier’sdescriptionofScopus(n.d.),itappearsthatanevolutionfromjournal-basedmetrics(IF)tobibliometricsbasedonindividualworks(articles,booksandbookchapters,conferenceproceedings)hasalreadytakenplaceinalargesectorofthemarket.ThisshiftaddressesamajortechnicalcritiqueofIF,usingjournalIFasasurrogateforarticleimpact.Elsevier’sScopusisthemajorsourceofdatafortheTimesHigherEducation(THE)andQSWorldUniversityRankings,amongothers.Assessmentofresearchandresearchersisoftenconductedbyresearchersthemselves,forexampleinpromotionandtenuredecisionsandinreviewofgrantapplications.Whileuniversityrankingssystemsaremovingtowardsarticle-levelmetrics,researchers’ownpracticesaredeeplyingrainedinacademiccultureandcontinuetorelyprimarilyonIF.Stephan,Veugelers&Wang(2017)discusswhattheycall“back-doorbibliometrics”,inwhichresearchersandreviewersreportand/orusejournalIFinassessmentevenwhenthisisnotrequired.Theyalsodiscusstheformaluseofbibliometricindicators,suchastheuseofrankingsderivedfromjournalIFinSpaininpromotionandsalaryincreasedecisions,andpaymentofbonusesinChinaaccordingtotheprestigeofthejournalinwhicha
5
researcherispublished.InsomeregionssuchasFlandersandBraziljournalIFisusedinallocatingresourcestouniversities.Whenresearchersfocusontheirownareasofspecialization,onemightassumethattheyhavethebackgroundknowledgetounderstandcommonlyusedmetrics.However,relianceonmeasuresandsurrogatemeasuresofjournalandarticleinfluenceiscommon,althoughresearchmetricsperseisnotacommonresearchspeciality.Tofullyunderstanduniversityrankings,weneedtoknowwhoproducesthedatathatfeedsintotherankings,andhowtheyproduceit.Elsevier’sScopusdataisthebasisfor38.5%oftherankingforTHE’sWorldUniversityRankings(WUR)(THE2018,p.82-83).Citationsorresearchinfluenceaccountfor30%oftheTHEWUR.Thesearebasedon“almost62millioncitationstomorethan12.4millionjournalarticles,articlereviews,conferenceproceedingsandbookandbookchapterspublishedover5years”.AccordingtoTHE,thesedata“helptoshowushowmucheachuniversityiscontributingtothesumofhumanknowledge…whoseresearchhasstoodout…beenpickedupandbuiltonbyotherscholars…”(p.83).THE’s“ResearchProductivity”isacountofthe“numberofpaperspublishedintheacademicjournalsindexedbyElsevier’sScopusperscholar,scaledforinstitutionalsizeandnormalisedforsubject”.Under“InstitutionalOutlook”,“InternationalCollaboration”isameasureofportionoftheuniversity’stotalresearchjournalpublicationsthathaveatleastoneinternationalco-author.ThetitlepaperoftheTHEreportstatesthattheworkis“inpartnershipwithElsevier”.QS’(2018)WorldUniversityRankingsusesCitationsperFaculty,astraightforwardcountofcitationstotheworksofscholarsattheuniversitybeingevaluated,usingtheScopusdatabase.Maclean’s(Dwyer,2017)addedbibliometricsindicatorsin2015,publicationsperfacultyandafield-weightedcitationimpactfactor,drawnfromScopus.Neworalternativemetrics-basedapproachesArticlemetricsisatpresentinaprocessofrapidevolution.Severalbasictrendsareobservable:ashiftfromjournaltoarticlelevelcitationmetrics,discussedabove;newtypesofmetricsoraltmetricsthatillustratedifferenttypesofindicatorsofusage,suchasviews,downloads,andsocialmediausage;andinclusionofmetricsandlinkstodownstreamcitingarticles,socialmedia,etc.,onpublisherwebsites.Inthissection,Iwillpresentabriefoverviewillustratingtherapidimplementationofdiverseapproaches,explaininplaintermswhatresearchersandpublishersinthisareaareaimingtoaccomplish,andarguethatwhilesomeaspectsofthesedevelopmentsareusefulforresearch,thereisaproblematiclackofcriticalreflectionontheimpactofthesedevelopments.In2012,Hausteinpublishedacomprehensivebookonthetechnicaldetailsandflawsofscholarlybibliometricsasofthatpointintime,concludingwitharecommendationforamultidimensionalapproachtometricstoovercometheflawsevidentinanyonemethod.Manyotherauthors,suchasKhodiyar,Rowlett,&Lawrence(2014),havesimilarlydiscussedthechangingnatureofassessmentofscholarlywork.Atpresent,thestateofpracticehasfaroutpacedscholarlyconceptionsofnewapproaches.Asdiscussedintheprevioussection,whileresearcherscontinuetoassumethatthejournalIFisstateoftheartinmetrics-basedevaluation,majoruniversityrankingsandtheworld’s
6
largestcommercialscholarlyjournalpublisher,Elsevier,havealreadymovedtoarticle-levelcitationmetricsusingdatafromtheirproductScopus.Meanwhile,publishingpracticealreadyreflectsheavyuseofneworaltmetricsthatincludebeyondcitations.Thestateofpracticecanbeeasilyobservedbybrowsingthewebsitesofscholarlyjournals.
Figure1.Elsevier’sJournalofDevelopmentEconomics.Source:https://www.journals.elsevier.com/journal-of-development-economicsOntheright-handsideofthehomepageofElsevier’sJournalofEconomicDevelopmentthereisameansforreaderstofilterarticlesthatincludestheoptions“mostdownloaded”and“mostcited”.Ontheleft-handsideofthepage,asillustratedinFigure1,thereisalistof5journalmetricsandalinkto“viewmoreonjournalinsights”;thispageincludesevenmoremetrics.Eachmetrichasanicon“I”formoreinformation;hoveringovertheiconbringsforwardthetechnicalexplanationforeachmetric.
Figure2:Trending.Source:NatureScientificReports.https://www.nature.com/srep/IfyouscrolldownthehomepageofNature’sopenaccessjournalScientificReportsyouwillseeasectioncalled“Trending”withtheword“Altmetric”prominentlydisplayedatthetoplefthandcornerasillustratedinFigure2.
7
Figure3.Articlemetricsfordinosaurarticle.Source:https://www.nature.com/articles/s41598-018-37943-3/metrics
Figure4.MentionsandTwitterdemographicsfordinosaurarticle.Source:https://www.nature.com/articles/s41598-018-37943-3/metricsClickingonthetoptrendingarticlebyaltmetric(dinosaurarticle)revealsmoredetailaboutthemetricsinvolved.AsofFebruary12,2019,thereare0citationsfromWebofScience.OnlineAttentionindicatestweeting,Facebook,andmediaattentionasillustratedinthefollowingFigures.Figure3showstotalcitations,onlineattentionandthealtmetricsscore.Figure4illustratesfurtherdetailthatcanbefoundbyscrollingdownfromFigure3,aclickablelistofmediareferencesandanoptiontoswitchtoscientificblogs,aswellasamapillustratingTwitterreferencesandalistoftweetingcountriesindescendingorderbynumberoftweets.Furtherdownonthepage(notillustratedhere)areexplanationsoftermsandsources.
8
TheseElsevierandNaturejournalseachreportneworaltmetrics,butnotthesameones.Why?Elsevierusescitationdatafromitsownproduct,Scopus.NaturePublishingGroupusescitationdatafromClarivate’sWebofScienceandCrossRefforScientificReportsratherthanScopusdata.Couldthisbebecausethetwocompaniesarerivals?NaturePublishingGroupownsSpringer,theworld’ssecond-largestcommercialscholarlypublisherandhenceamajorcompetitorforElsevier.PerhapsNatureprefersnottodisplayElsevier’sScopusdata,oritmightbethatElsevierprefersnottoprovideNaturewithareasonablepriceforuseofScopusdatatoacompetitor.PublicLibraryofScience(PLOS)providesadetailedexplanationoftheirarticle-levelmetrics(ALMs)onthePLOSwebsite:https://www.plos.org/article-level-metrics.PLOSdefinesALMSas:“quantifiablemeasuresthatdocumentthemanywaysinwhichbothscientistsandthegeneralpublicengagewithpublishedresearch”.SuggestedusesofALMsforresearchersaretocommunicateimpactingeneralandtofunders,toraisearesearcher’scareerprofile,andtofindcollaborators.
CallsforchangeinresearchassessmentDORAThefirstmajorcalltoactionisthe2012SanFranciscoDeclarationonResearchAssessment(DORA)initiatedbytheAmericanSocietyforCellBiology.DORA'srecommendationsstate"theneedtoeliminate[emphasisadded]theuseofjournal-basedmetricssuchasJournalImpactFactorsinfunding,appointment,andpromotionconsiderations".AsofOctober2018,DORAhasbeenendorsedbyover600organizationsand13,000individuals,myselfincluded.DORAdoesnotquestiontheconceptofmeasurementperse,statingthat:“fundingagencies,institutionsthatemployscientists,andscientiststhemselves,allhaveadesire,andneed,toassessthequalityandimpactofscientificoutputs.Itisthusimperativethatscientificoutputismeasuredaccuratelyandevaluatedwisely”.ImplicitinDORAisanassumptionthatthepeer-reviewedjournalarticlewillcontinuetobethemostfrequentmeansofdisseminationofnewknowledgeintheforeseeablefuture.Developmentofmetricstoincludenewformsofresearchoutputssuchasdatasetsandsoftwareisencouraged.DORAincludesgeneralrecommendationsandspecificrecommendationsforfundingagencies,institutions,publishers,organizationsthatsupplymetrics,andresearchers.Thelatter,wheninvolvedincommitteesmakingdecisionssuchashiring,promotion,andtenure,areencouragedtomakeassessmentbasedonscientificcontentratherthanpublicationmetrics.Oneofthedeficienciesofallcitation-basedmetricsnotedinDORAisaskewtowardsreviewarticlesasauthorstendtocitereviewarticlesratherthantheoriginalworksthatarereviewed.Publishersandresearchersarecalledupontoencourageauthorstociteoriginalresearch.Itisrecommendedthatfundingagenciesandinstitutions“considerabroadrangeofimpactmeasuresincludingqualitativeindicatorsofresearchimpact,suchasinfluenceonpolicyandpractice”.However,publishersareencouragedto“makeavailablearangeofarticle-levelmetrics”andresearchersto“usearangeofarticlemetricsandindicators”,whichsuggestsadeeperquantitativeratherthanaqualitativeapproach.Publishersareaskedto“encourageresponsibleauthorshippracticesandtheprovisionofinformationaboutthespecificcontributionsofeachauthor.”Qualitativeinformationabout
9
authorcontributionswouldappeartoadvancethepotentialforenhancedqualitativeassessment.LeidenManifestoAgroupofself-describedscientometricians,socialscientistsandresearchadministratorshighlightsomeofthemajorissueswithjournalIFandothermetricsandlist10principlestoguideresearchevaluation,principlesthatwerecrystallizedatthe19thAnnualConferenceonScienceandTechnologyConferenceIndicatorsheldinLeidenin2014;hencethislistisreferredtoastheLeidenManifesto(Hicksetal.,2015).TheLeidenManifestohttp://www.leidenmanifesto.org/hasbeentranslatedinto18languages.Thefirstprinciplestates:“quantitativeevaluationshouldsupportqualitative,expertassessment”.Theauthorscitean“impactfactorobsession”,stating“soaringinterestinonecrudemeasure–theaveragecitationcountsofitemspublishedinajournalinthepasttwoyears–illustratesthecrisisinresearchevaluation”.CritiqueofmetricsisnotlimitedtoIF.Simplepublicationcountscanbeproblematicaswell.AnexampleisprovidedofarelativelylowratingofagroupofEuropeanhistoriansinanationalpeer-reviewexercisesimplybecausehistorianstendtowritebooksratherthanjournalarticles.H-factorincreaseswiththeresearcher’sage,eveniftheresearcherdoesnotproducenewpapers.Theh-factorisalsodatabase-dependent.Computerscientistscanhaveanh-factorof10inWebofSciencebut20–30inGoogleScholar.Precisionmatters;“asinglehighlycitedpublicationslightlyimprovesthepositionofauniversityinarankingthatisbasedonpercentileindicators,butmaypropeltheuniversityfromthemiddletothetopofarankingbuiltoncitationaverages”Theauthorssuggestthatrelyingonasinglemeasurewill“invitegamingandgoaldis-placement(inwhichthemeasurementbecomesthegoal)”,andoneoftheirproposedsolutionsismultiplemetrics.LeidenPrinciple2,isto“Measureperformanceagainsttheresearchmissionsoftheinstitution,grouporresearcher”.Thereisnosinglemetricthatmakessenseineveryresearchcontext.Forexample,consideranactionresearchprojectdesignedtohelpacommunitygroupaddressanissueofconcerntothem.Ideally,designoftheproject’sgoalsandevaluationmeasuresshouldbeundertakeninconsultationwith,orledby,thecommunitygroup.Theoptimalmeasuresofsuccesswouldprobablybereal-worldindicatorsofchange.Homelesspeoplewanthomes,notcitationstoarticlesabouthomelessness.Publicationofresults,writteninplainlanguage,invenuesthatarephysicallyandintellectuallyaccessibletothecommunity,suchasthecommunity’sownnewsletterorblogoralocalworkshopmaybemoreeffectiveinmeetingthegoalsoftheresearchthanpublicationinscholarlyjournalsusingacademicjargonthatthegroupmaynothaveaccesstoorunderstand,orpresentationsatscholarlyconferencesthatgroupmemberscannotaffordtoattend.Thesameprincipleswouldapplytoacademic/industryandacademic/governmentcollaborations.Thisisnottosaythattraditionalacademicideasofexcellencedonotapply,ratherthatmeasuringexcellencebynumberofpublicationsandcitationsinprestigiousjournalsisnottheoptimalwaytoevaluateeverytypeofresearchproject.LeidenPrinciple3states:“protectexcellenceinlocallyrelevantresearch”.OneexampleoftheproblemisthatofSpanishlawwhichstatesthedesirabilityofSpanishscholarspublishinginhigh-impactjournals.Insociology,thehighestimpactfactorjournalsarepublishedinEnglishintheUnitedStates(US);likelyasaresultofthis,highlycitedSpanishsociologistsarethosewhofocusoneitherabstractmodelsorUSsocialproblems.
10
ScienceEurope’sNewvisionformoremeaningfulresearchassessmentInJuly2017,ScienceEurope(2017a),anon-profitorganizationbasedinBrusselsrepresentingmajorresearchorganizationsacrossEurope,issuedapositionstatement“onanewvisionformoremeaningfulresearchassessment”.Thepreamblecontraststhebroadimpactofresearchonsociety,oftengainedthroughagradualdevelopmentofnewknowledge,withmetricsdesignedtomeasuretheimpactofaspecificstudy,andpointsoutthatitisnotalwayspossibletoconnectsocietalimpactwithaparticularresearchstudy.Thepreamblegoesontodiscusstheconceptofthevalueofresearch,abroadernotionofresearchimpactthatincludessocietalvalues.Societalprogressdrawsfrombothresearchoutputsandothersources;toonarrowanemphasisonconcreteimpactmaygenerateunintended,andnotnecessarilybeneficial,effectsonresearchactivity.Theconclusionstates“thereisgreatdiversityinthewaysinwhichresearchbringsitsimmensevaluetosociety.Someofthesewaysareindirectorintangibleandcannoteasilybemeasuredbystrictlydefinedimpactassessmentcriteria.Othersarelong-termorunpredictableandmaynotyetbevisibleatthetimethattheresearchisevaluated...ultimately,thebestwaytomaximisethevalueofresearchtosocietyisbyensuringthattheresearchproducedmeetsthehigheststandardsofqualityandexcellence”.ScienceEurope’sstatementemphasizestheimportanceoftrustbetweenresearchersandsociety.ThisisreflectedinPriority2ofScienceEurope’s(2017b)TaillinnCallforAction,whichcallsforresearchorganizationsandfundersto“recogniseabroadnotionofimpactthatacknowledgesthesocietalvalueofresearchforpolicyandpractice”andcallsonpolicymakers,researchfunders,andacademicto“fosterthenecessaryculturalchangetoembracethebroadnotionofimpact”.Implementationofchangeinapproachestoresearchevaluationisinanearlystage.InSeptember2018theEuropeanCommissionendorsedPlanSandCoalitionS,anambitiousplantoacceleratethetransitiontoopenaccesspublishing;“thewayweevaluateresearchoutputs”isidentifiedasoneofthebarrierstochange(EuropeanCommission,2018).ScienceEurope’sPresidentMarcSchiltz(2018)inastatementcalledWhyPlanS,statesthat“Wecommittofundamentallyrevisetheincentiveandrewardsystemofscience,usingtheSanFranciscoDeclarationonResearchAssessment(DORA)asastartingpoint”. The basic idea is for all funding agencies, particularly Europe but the aspirations are to inspire change globally, to commit to changing how research is evaluated, from traditional to new or altmetrics. It will be interesting to observe progress towards implementation of this ambitious plan over the next few years.
DiscussionMetricsversusqualityinresearchCanmetricsystemscapturetheessentiallyqualitativenatureoftheconceptofqualityinresearch?ThemaingoaldrivingdevelopmentofcitationindexingasdescribedbyGarfieldin1955wassothatresearcherscouldtrackforwardsfrompublishedworktocitingworksthatmightpointoutcritiqueoftheoriginal.Iarguethatthewholeideaofusingmetricstoassessthequalityofresearchandresearchersisrelativelynewandhasnotreceivedthecriticalattentionthatitdeserves.Thissectionaimstobeginaprocessofapplyingscholarly
11
critiquetothisarea.AsDORAnotes,thereisaskewincitationstowardsreviewarticlesratherthanoriginalresearch.Thisraisesaquestion:ifresearchersarecitingreviewarticlesratherthanoriginalresearch,aretheyevenreadingtheoriginalresearch,nevermindtrackingdownstreamcitations?Ifweassumethatreviewarticlescontainalltheimportantinformationfromeveryreviewedarticle,captureitaccurately,andthattheoriginalarticlesarenevercritiqued,retractedorrefutedbysubsequentresearch,thenreadingreviewarticlesisnotproblematic,butthenwouldbenoneedforcitationlinkingtouncovercritique.Currentapproachestoresearchassessmentassumethatwhenitcomestoscholarlypublishing,moreisbetter.Giventheconstantincreaseinthevolumeofproductionofscholarlyworksandtheavailabilityofcitationindexingtopermitmorecarefulchecking,shouldn’tresearcherstodaybespendingrelativelymoretimereadingratherthanpublishing?Iftheywere,wouldn’ttheybepublishinglessratherthanmore?Assessmentsystemsbasedonthepremisethatmoreisbetterseemlikelytoriskincreasingerrorssuchasinvalidresults.Researchintocurrentpracticewouldbehelpful.Onemightsurveyresearchersonwhethertheyactuallyreadalloftheworksthattheycite,whethertheyrelyonsecondarysourcessuchasreviewsorgototheoriginals,andwhethertheyusecitationindexestocheckdownstreamcitingsources.Or,onemightanalysewrittenpublicationstoseewhetherthereareerrorsthatmighthavebeencaughtwithmorein-depthreading.Advancingourknowledgerequiresquestioningunderlyingassumptionsinadditiontobuildingonexistingwork.Twoassumptionsintheareaofresearchassessmentthatshouldbechallengedaretheassumptionthat“impact”itselfisnecessarilypositivelycorrelatedwithgoodqualityworkandthatimpactisinherentlydesirable.ThesecondmosthighlycitedretractedpaperaccordingtotheRetractionWatch(n.d.)blogistheinfamous1998paperbyWakefieldetal.publishedinthehighlyprestigiousjournalLancet,purportingtomakeaconnectionbetweenvaccinationandautism.Thisarticlehasbeencitedover1,000timesinthelistofjournalsincludedinWebofScience,with640citationsbeforeretractionand468citationsafter.Anyoftheexistingoremergingmetrics-basedapproachestoresearchassessmentswouldfindthatthisstudyhashadalotofimpact.ThearticlewaspublishedinahighIFjournal.Itisahighlycitedarticle,whichwouldresultinhigharticle-levelrankingsandboosttheh-factorofalloftheauthors.Ifweconsiderreal-worldimpact,theinfluenceofthisarticleintheanti-vaccinationmovementandthesubsequentreturnofdiseasessuchasmeaslesdemonstrateanexceptionalreal-worldimpactforasinglearticle.Thisillustratesthedangerofassumingthatimpactisnecessarilygood.Likealmostallqualitiesofthingsintherealworld,impactisneithergoodnorbadinandofitself,butrathermustbeinterpretedincontext.Ifuniversitiesandresearchfundersarerelyingonuniversityrankings,thisprovidesanadditionalincentivetofocusontraditionalformsofpublication.Forexample,thebibliometricspartnerofTimesHigherEducation(2018)inproducingtheWorldUniversityRankingsisElsevier.Tomeasureresearchproductivity,Elseviercounts“thenumberofpaperspublishedintheacademicjournalsindexedbyElsevier’sScopusdatabaseperscholar”.Tocalculatecitationsorresearchinfluence,Elsevier“examined…citationstojournalarticles,articlereviews,conferenceproceedingsandbooksandbookchapters”.Thiscountofcitationstoparticulartypesofworksisadefactoendorsementofthesetypesofworks.ItisprobablynotacoincidencethatElsevierisahighlyprofitablepublisherofexactlythesetypesofworks.
12
Myownexperienceasascholarconfirmsthisfocusonalimitedrangeofformats.Ouruniversity’sonlineCVsystemislikelytypicalincategorizingtypesofpublications–booksandbookchapters,peer-reviewedjournalarticles,non-peer-reviewedarticlesandsoforth.Thiscategorizationisunderstandableforhistoricalreasons,howevertheendresultisthatthemajorityofmyworks,andalmostallthatIconsidermymostimportantandleading-edgeworks,suchastheopendatadiscussedaboveandmyscholarlyblogs,SustainingtheKnowledgeCommonsandTheImaginaryJournalofPoeticEconomicsarelabelledas“other”andwouldcountforlittleornothingunderexistingmetrics-basedassessmentapproaches.RetractionWatchhttp://retractionwatch.com/bloggerstrackandreportonpublishedarticlesthatwereretractedafterpublication.UnlikeJCR,InCites,orScopus,RetractionWatchaddressestheoriginalmaingoalofcitationindexesasproposedbyGarfieldin1955,“abibliographicsystemforscienceliteraturethatcaneliminatetheuncriticalcitationoffraudulent,incomplete,orobsoletedatabymakingitpossiblefortheconscientiousscholartobeawareofcriticismsofearlierpapers”.Mostretractionsreflecterrorsdiscoveredafterpublicationbutalsooccasionallyfraudisuncoveredandreported.RetractionWatch’s(n.d.)top10mosthighlycitedretractedpaperslists10papersthathavebeencitedmorethan550timesinjournalsindexedinWebofScience(asofJanuary2018),andthislistincludes2papersthathavebeencitedmoreoftenafterretractionthanbefore.Theblogalsotracksevidencethatcitationofretractedpapersisanongoingproblem.RetractionWatchdemonstratesafundamentalflawwithcurrentapproachestoresearchassessment,whichfocusesontheimpactofscholarlywork,whethermeasuredindirectlythroughmetricssuchasjournalIFordirectlythrougharticle-levelmetrics,andforthemostpartneglectsthemoreimportantquestionoftheaccuracyofscholarlywork.BibliometricsandtheeconomicsustainabilityofscholarlycommunicationIncreasingmarketconcentrationwasthesubjectofinvestigationbytheUnitedKingdom(UK)OfficeofFairTrading(2002).AsdescribedbyMorrison(2012),industryconcentrationandgrowingprofitsofafewlargecommercialscholarlyjournalpublisherswasaccompaniedbyasignificantdecreaseintheaveragenumberofcopiesofscholarlymonographsproducedandsold.Recently,LaRivière,Haustein&Mongeon(2015)reportanincreaseinconcentrationinthescholarlypublishingmarket,withthetop5publishersaccountingformorethan50%ofthearticlesindexedinWebofScience.TheScholarlyPublishingandAcademicResourcesCoalition(SPARC)(n.d.)maintainsalistofjournalbigdealcancellationsbyuniversitylibraries,librarygroupsandstatelibrarysystems,andnationalcoalitionssuchastheConsortiumonCoreElectronicResourcesinTaiwan(CONCERT).Thisisnotahealthysystem;thehighpricesandprofitsofafewcommercialscholarlypublisherscannotbesustainedbyacademiclibraries,andtheeconomiccloutbehindthebigpackagedealsresultinlittlefundingleftoverforpublishingscholarlymonographsandsupportingthejournalsofsmallerpublishers,particularlyinthehumanitiesandsocialsciences.Myresearchsuggeststhesametrendofcommercialconcentration,involvingthesamecompanies,isemerginginopenaccesspublishing.Asof2017,thelargestopenaccessjournalpublisherbynumberofjournaltitleswasSpringerNature(includingBioMedCentral),followedbyElsevier(Morrison,2017).Thisdevelopmentisnotananomaly.Identifyingasub-setofjournalsas“core”andthereforemoredesirabletopublishinandmoreessentialtopurchaseincreasestheir
13
marketvalue.“Core”isinquotestoemphasizethatthisisanessentializationoftheconceptformarketpurposes.Theprioritiesoffor-profitpublishersarereturningprofittoshareholdersorprivateowners,notthehealthofthescholarlypublishingecosystem.Thus,itislogicalthatjournalIFexacerbatestheproblemofaffordabilityofscholarlypublishingandsimilarlylogicaltohypothesizethatnewbibliometrics-basedapproacheswillhaveasimilareffect.Qualitativefocusedassessment:how,why,andtheUniversityofOttawaasmodelTherearemodelsforassessingresearchatthelevelofevaluationofindividualresearchers,programs,andinstitutionsthatexemplifyanunderstandingofthebroadervalueofresearchtosocietyandaddressthecomplexityofthediversityofresearchasexpressedintheScienceEuropevisionandtheLeidenManifesto.Atmyownuniversity,theUniversityofOttawa,thecriteriaforevaluatingfacultymembersforpromotionandtenureiscollaborativelydevelopedbyfacultyandadministrationandgovernedbytheAssociationofUniversityProfessorsoftheUniversityofOttawa(APUO)CollectiveAgreement(2016-2018).ThefulltextofallsectionsofthecollectiveagreementdirectlyrelevanttoresearchassessmentcanbefoundintheAppendix.ThelanguageoftheAPUOagreementaddressesthequestionofdiverseandevolvingformsofscholarlyworks.Section23.3.1(h)regardingthetypesofmaterialthatmembersmaysubmitforassessment,states“itisunderstoodthatsincemethodsofdisseminationmayvaryamongdisciplinesandindividuals,disseminationshallnotbelimitedtopublicationinrefereedjournalsoranyparticularformormethods”;oneexampleofotherformsislistedinsection(a),“inthecaseofliteraryorartisticcreation,originalworksandformsofexpression”.Canada’sSocialSciencesandHumanitiesResearchCouncilrecognizes“research-creation”asavalidformofdissemination.Typically,anewfacultymemberattheUniversityofOttawaishiredattherankofAssistantProfessorand,aftersixyears,appliesfortheseniorrankofAssociateProfessor,apromotionthatautomaticallyinvokestenure.ThecriteriaforresearchassessmentatthisstagearecoveredundertheAPUOCollectiveAgreementSection25.3.2.2(c).Toachievetenure,anewfacultymembermustdemonstrateproductionofgoodqualityscientific,literary,artistic,orprofessionalworks,thatgobeyondworkdoneinthecompletionofthedoctorateandthatshowcontinuousprogress.Evaluationisconductedbythreeoutsideevaluatorsandreviewedbycommitteesatthefacultyanduniversity-widelevel.AsimilarprocessisfollowedwhenafacultymemberappliesforpromotiontotherankoffullortitularProfessor.Thisisaholisticcareer-levelpeerreviewprocess.Ironically,andsomewhatmysteriously,inspiteofthisqualitativeandinclusiveapproachtoassessmentofresearchandresearchersratherthansimplisticmetrics,theUniversityofOttawadoesverywellinmetrics-basedrankings.AccordingtotheUniversityofOttawa&GovernmentofOntarioMinistryofAdvancedEducationandSkillsDevelopment2017-2020StrategicMandateAgreement(p.17),“IndependentnationalandInternationalrankings(suchasResearchInfosource,QSWorldUniversityandtheTimesHigherEducation)consistentlyplaceuOttawaamongthetopthreeOntariouniversities,amongCanada’stop10researchuniversitiesandamongthetoptwopercentoftheworld’suniversities”.Whyisthis?Theanswertothisquestionisnoteasytoascertainbyreviewingrankingagencies’descriptionsofmethodology.TimesHigherEducationclaimsforthe2019
14
rankingstohavethefirstauditeduniversityrankingsandappearstobethemosttransparent.TheoverallrankingfortheUniversityofOttawafrom2012to2019variedfromahighof171(2013)toalowof251-300in2017,risingto201–250in2018and176in2019.TheareaswheretheUniversityofOttawaappearstoscorerelativelywellarecitations,industryincome,andinternationaloutlook.Citationscountfor30%oftheoverallweightandsoarelikelythemajorfactor.ThereisnoobviousreasonfromthedetailedmethoddescriptionofTHEastowhytheaveragecitationstoresearchpublishedbyUniversityofOttawafacultywouldhavechangedduringthistimeframe.Overthistimeframe,thefull-facultyfacultycomplementhasdecreasedslightly;overtime,onewouldexpectthistodecreasethenumberofcitationswithlessresearchbeingproduced.Aslightdecreaseinthefacultycomplementwouldaccountforaslightlyhigheraveragecitationrateifthefacultycomplementwerefactoredin,howevertheTHEmethodologydoesnotstatethatthisisthecase.Industryincomemayreflectlocaleconomicconditions.TheUniversityofOttawa’scentrallocationinthenation’scapital(closetodowntown,CityHall,Parliament,surroundedbyembassies,closetonationalcorporateheadoffices)maybeafactorintheuniversity’sstrengthsinindustryincomeandinternationaloutlook.However,thewidevariationinoverallstandingoverrecentyears,givenastableuniversity,suggestthatchangesinhowrankingsarecalculatedarealargefactorincurrentstandings.AttheUniversityofOttawa,regularcomprehensiveassessmentprocessesthatincludepeerassessmentoffacultyandstudentresearcharealreadyinplaceattheprogramlevel.Atminimum,eachprogramundergoesaprovinciallymandatedcyclicalreviewevery7years.Thisisfarmorein-depthassessmentthancountingpublicationsorcitations.Forexample,externalreviewersconductin-personinterviewsseparatelywithfaculty,students,andadministratorsandhavetheopportunitytoaskquestionsnotjustaboutresearchoutputs,butalsoaboutinstitutionalsupportforresearchintermsoftime,facilities,assistancewithgrantapplicationsandsoforth.Inaddition,someprograms,particularlyprofessionalprograms,undergoprofessionalaccreditationprocesses,thatalsoreviewresearchundertakenatthedepartmentallevel.Forexample,theSchoolofInformationStudiesundergoesarigorousaccreditationprocesscoordinatedbytheAmericanLibraryAssociation.Forauniversitywiththesein-depth,holisticresearchassessmentpracticesalreadyinplace,aturntogreaterrelianceonsimplisticmetricsbasedonalimitedandbackwards-lookingunderstandingofformatsandwhatconstitutesgoodqualityscholarlyworkwouldbeastepbackwards.TheprimarymissionofElsevierisreturningprofittoshareholdersofitsparentcompanyRELX;theprimarymissionofClarivateisreturningprofittoitsprivateowners.ThemissionoftheUniversityofOttawa(2017-2020),withrespecttoitsroleasaresearch-intensiveuniversity:“weprovideourstudentswithanoutstandingeducationandenrichtheintellectual,economicandculturallifeofCanada,helpingourcountryplayanimportantandvaluedroleamongthenationsoftheworld”.Themissionofeachuniversitywilldifferslightly,butwilltendtorevolvearoundthecentralfunctionsofteaching,researchasanactivitydesignedtofurtherourcollectiveknowledge,andservicetotheacademyandtosocietyasawhole.Iarguethatweshouldtrustscholarsandtheacademytodesignandimplementassessmentmechanismsthatreflectandprioritizeourgoals(missions,vision),notthoseofoutsidepartieswhoseprimaryinterestsareinherentlydifferentfromourown.
15
Conclusion:theirrationalrationalityofmetrics-basedassessmentofresearchItislogicalforpeopletowanttomeasureprogresstowardsthegoalsthatwedesire.Manymeasuresarevalidandlogical.However,whenwefocusonthemeasuresperseratherthanthegoals,wecanendupwithresultsthatdonotachieveourgoals.ThisiswhatIcallasuperficiallyrational(mathematical,calculating)approachthatisactuallyirrationalintermsofwhatweareactuallyattemptingtoachieve,orirrationalrationality.Currentandemergingformsofmetrics-basedassessmentofresearchandresearchersdisplaymajorproblemswithirrationalrationalitythatcreateincentivesthatarenotcompatiblewithagoalofproducinganddisseminatingqualityresearch.Theseproblemsmeriturgentattentionbeforeourcurrentfixationwithmetricsfurtherentrenchesexistingproblemsandneworaltmetricsintroducesnewones.Therearevalid,logicalreasonsforuseofsomemetricsinassessingresearch,researchers,researchinstitutions,andpublishers.Auniversityshouldbeabletopointtoasubstantialcorpsoffacultywitharesearchmandateandabodyofresearchworksproducedbyitsfacultytocallitselfaresearchuniversity.Anindividualresearchershouldbeabletopointtoacollectionofpublishedworksand/orsubstantivework-in-progressinordertobeconsideredaproductiveresearcher.Individualresearchersandresearchteamsmayfindithelpfultodevelopspecificmeasurablegoalsthatmakesensefortheirownprojects.Journalsandotherpublisherscanusemetricstoassessmarketingefforts.Bibliometricsisausefulresearchmethodforgeneratingnewknowledge.However,justbecausesomemetricsarehelpful,itdoesnotfollowthatubiquitousmetricsarehelpful.Abitofsaltaddsflavortofood;excesssodiumcauseshighbloodpressure,increasingtheriskofheartattackorstroke.Evaluationbasedonmetricslooksscientific,doesn’tit?Numbersareobjective.Metrics-basedevaluationisrationalandcalculating;scientistsoftenuselotsofdata.However,theresemblanceissuperficial.Logicisapowerfultool;butthevalidityoflogicalargumentsdependsonthevalidityoftheunderlyingassumptions.Inordertoassesswhetherwearemakingprogressinscience,weneedtounderstandhowscienceworks.Asdiscussedabove,ourcurrentapproachestosciencearecompatiblewiththeproductionofdangerouslyerroneous“facts”suchastheequationofvaccineswithautismandfalsebeliefinthesafetyofdrugs;irreproducibleresearch;and,atendencytociteliteraturereviewsthatmakesmewonderhowoftentheoriginalstudiesareactuallyread.Ifthisisthesituationinscience,whataboutotherbranchesofknowledge?AsCamic,Gross&Lamont(2011)discuss,whiletherehasbeensomereflectiononpracticeinthesciencessincepublicationofKuhn’s(1962)TheStructureofScientificRevolutionsinthe1960’sandsubsequentdevelopmentofthefieldofscienceandtechnologystudies(STS),parallelstudyofprocessesintheareaofthesocialsciencesisjustbeginning.Ifwedonotevenknowwhatscholarsinthesocialsciencesdo,howcanweclaimtoknowhowtomeasurewhethertheyaredoingitwell?Thestudyofphilosophyandpracticeofscience,whilemoreadvancedthanthestudyofsocialsciences,raisesmorequestionsthananswersaboutmetrics-basedapproaches.Forexample,itislogicaltoassumethattheparadigmshiftsdescribedbyKuhn(1962)willleadtosituationswherewhetherworksarecited,andbywhom,dependsonthephaseofdevelopmentofnewideas.Onemighthypothesizethatworksthatfitacurrentparadigmwillbecitedmorethanpioneeringworks,inwhichcaseusingcitationstoassessthevalueofresearchwilltendtoincentivizeconservatismoverinnovation.
16
IarguethatKuhn’sworkitselfillustratestheproblem.ThomasKuhnwasayoung,white,highlyeducatedmalebasedintheUnitedStateswhowroteaboutscientificrevolutionsinthe1960’s.Thisworkwaswidelyread,studied,andcited,withinashorttimeafterpublication.Anearlierwork,Fleck’s1930’sground-breakingGenesisofascientificfact(Fleck,1979),didnotenjoythisimmediateacclaim.Intheforewordtothe1979edition,Kuhndescribesfindingthiswork,writteninGerman,byhappenstancewhilebrowsinginthestacksofalibrary.FleckwasaJewishintellectuallackingformalcredentialswhoseworkwaspublishedinGermanyinthe1930’s.UnlikethepopularreceptionofKuhn’swork,only600copieswereprintedofFleck’swork,only200weresold,andonly6weredeliveredtotheUnitedStates.Kuhn’sphilosophywasinspiredbythiswork,butthisdoesnotrendertheworkobsoleteasKuhn’sideascomplementratherthansupersedethoseofFleck.Forexample,whereKuhnemphasizedsuddenrupturesinscientificthinking,Fleckemphasizedcontinuityofbasicpremisesinapparentlyrevolutionaryadvancesinknowledge.Whatdoesthishavetodowithuniversityrankings?Ihavedescribedadeeplyflawedsystem,withanillusoryappearanceofscientificbasis,thatincentivizesquantityofproductionofresearchworksoverquality,noveltyoverrigorouscritiqueandreplication.Themetricsbehindthissystemfeedintouniversityrankings,andtherankingsreinforcethistrendtowardirrationalrationality.Thecurrenttrendtowardsneworaltmetricswillcreateevenmoreirrationalrationality.Itislogicaltoexpectthatthesenewmetrics,particularlymetricsthatdonotdependonacademiccitations,willamplifyexistingproblemswithmetrics-basedevaluationand/orcreatenewones.Ipredictthatsuchmetricswillreflectpre-existingsocialbiases.Theextenttowhichindividualworksarecited,downloaded,and/orsharedviasocialmediaarelikelytocorrelatewithgenderandethnicbiasesaswellasthepopularityoftopicsstudied.Inaddition,metricsthatdonotdependonacademiccitations(downloads,tweets,etc.)arefarmorevulnerabletodeliberatemanipulation.Thefossilfuelindustrycanaffordtohirepeopletodownloadandtweetevidenceofclimatechangedenial,forexample.Anotherfactorthatshouldbeconsideredbeforeusingsuchdataasasurrogateforqualityofresearchistheimpactthatusageofsuchmetricscouldhaveontheresearchitself.Forexample,ifcancerresearchersfindithelpfultousesocialmedia,theycanandshoulddoso.Butifmetricsbasedonnon-academicuseweretoformthebasisofassessmentandresearchinfuture,thiscouldresultinaredirectionofeffortsfromcancerresearchtosocialmediasharing.Asfortheimpactofpopularityoftopicsinnewmetrics,onacynicalnoteItakecomfortinthepossibilitythatsomedayImayhavereasontomoveforwardwithastudyalongthelinesof“correlatesofperceivedattractivenessofjuvenilefelinesonYouTube”[academeseforhowcomethosekittycatsonYouTubearesodarncute]toprovemyworthinessasaresearcher.Onaseriousnote,itismyexperienceasalongtimepractitionerofopenresearchthatthepopularityofmyworksdoesnotcorrelatewiththeimportanceofitscontribution.Myground-breakingbookchapter“Theimplicationsofusagestatisticsasaneconomicfactorinscholarlycommunications”beginssomeofthediscussionthatcontinueswiththischapterandintroducesimportantbutcounter-intuitiveideas(Morrison,2005).ThisworkdoesnotenjoyevenasmallfractionofthesocialmediapopularityofmyDramaticGrowthofOpenAccessblogseries,designedtosupporttheadvocacyeffortsofaglobalmovement.
17
Theevolutionofresearchmetricsdescribedinthischaptercapturestheirrationalrationalityofmetrics-basedresearchassessment.Inthe1950’sand1960’swedevelopedtoolstohelpresearchersandlibrariescopewiththeever-expandingvolumeofscholarlyliteraturebyconnectingcitingandcitedworksandidentifyinghighlycitedjournalstohelplibrarieswithdecisionsaboutpurchasesandcancellations,andresearcherswithdecisionsaboutreadingandpublishing.Theresultingmetric,IF,becameayardstickforevaluatingtheworthinessofresearchandresearchersevenwhenitwasacknowledgebyexpertsandtheproducingcompanythatthismetricwasnotatallsuitableforthispurpose.AttemptstoaddressthetechnicalflawsofIF(connectedtojournalsratherthanarticles)areaddinganewlayerofmetricsbasedoncitationstoindividualworksthatrankingcompaniesarealreadyincorporatingintoassessmentofuniversitiesandthatarebeingmarketedasameansofassessingresearchers.Theprofitgoalsofmetricsbasedcompanies(scholarlypublishers,citationmetricsandrankingservices)areovertakingtheresearchmissionsofuniversities.Wedeveloptoolstohelpusachieveourgoals,thenwebecomeslavestothetools.Thatisirrationalrationality.Inthefuture,ifwecontinueonthecurrenttrajectory,weshouldexpectanadditionallayerofmuchmoreillogicalmetrics-basedcontrolofresearchandresearchersintheformofaltmetricsbasedonusagebeyondtheacademy.Ifexcessrelianceonmetricsistheproblem,whatistheremedy?Let’sdevelopandusemetricswheretheymakesense,basedonthegoalsofindividualresearchprojectsandinstitutions.Butlet’sdosowithagrainofsaltandnotrelyonmetricswheresuchrelianceisnotscientificandmaybecounter-productivebycreatingperverseincentivesforquantityandnoveltyoverqualityandfavourparticularformats,evenastheyarebecomeobsolete.Howcanweimplementthisremedy?Wecanuseapproachesthatappropriatelyweightqualityandthatrecognizethediverseformsofresearch.Insteadoftranslatingaresearchdossierintoun-scientificmetrics,readandreviewtheworks.Thisisn’tnew,andshouldn’tbehard.Thisiswhatwedonowwhenweassessathesisorpeer-reviewtheworksofotherresearchers.AttheUniversityofOttawa,wehaveacollectiveagreementthatacknowledgesthediversityofresearchanditsproducts;detailsareintheAppendixifanyonewouldliketoconsiderthisasonepotentialmodelforchange.ReferencesAPUOCollectiveAgreement(2016-2018).RetrievedJanuary25,2018from
http://www.apuo.ca/collective-agreement/collective-agreement/AssociationofResearchLibraries(ARL).(1989).ReportoftheARLserialspricesproject:A
compilationofreportsexaminingtheserialspricesproblem.Washington,DC:TheAssociationofResearchLibraries.RetrievedJanuary15,2018fromhttp://catalog.hathitrust.org/Record/001527850
Baker,M.(2016).1,500scientistsliftthelidonreproducibility.Nature533,452–454
(26May2016)doi:10.1038/533452aBruckner,Till,&Ellis,Beth.(2017).Clinicaltrialtransparency:akeytobetterandsafer
medicines.RetrievedAugust1,2017fromhttps://media.wix.com/ugd/01f35d_0f2955eb88e34c02b82d886c528efeb4.pdf
Camic,C.;Gross,N.,Lamont,M.(2011).Socialknowledgeinthemaking.Chicago:The
UniversityofChicagoPress.
18
Casadevall,A.&Fang,F.(2010).Reproduciblescience.InfectionandImmunity78:12.
doi:10.1128/IAI.00908-10.RetrievedMay2,2017fromhttp://iai.asm.org/content/78/12/4972.full
ClarivateAnalytics(n.d.).Products:JournalCitationReports.RetrievedJanuary15,
2018fromhttps://clarivate.com/
Dwyer,M.(2017).Measuringexcellence:howwerankCanada’suniversities.Maclean’s.RetrievedJanuary23,2018fromhttp://www.macleans.ca/education/measuring-excellence-how-we-rank-canadas-universities/
ElsevierScopus(n.d.).RetrievedJanuary23,2018fromhttps://www.elsevier.com/solutions/scopus
European Commission (2018). Plan S and cOAlition S – accelerating the transition to full
and immediate Open Access to scientific publications. (Statement). Retrieved Feb. 18, 2019 from https://ec.europa.eu/commission/commissioners/2014-2019/moedas/announcements/plan-s-and-coalition-s-accelerating-transition-full-and-immediate-open-access-scientific_en
Fleck, L. (1979.). Genesis and development of a scientific fact. (T. Trenn & R.K. Merton,
Eds; T. Trenn & F. Bradley, Trans.) Foreword by Kuhn, T.S. Chicago : University of Chicago Press.
Garfield(1955).Citationindexestoscience:anewdimensionindocumentationthrough
associationofideas.Science.1955:122:108-111.RetrievedJanuary15,2018fromhttp://garfield.library.upenn.edu/essays/v6p468y1983.pdf.
Garfield,E.(2006).Thehistoryandmeaningofthejournalimpactfactor.Journalofthe
AmericanMedicalAssociation295:1.
Guédon,J.C.(2001).InOldenburg'slongshadow.Washington,D.C.:AssociationofResearchLibraries.
Haustein,S.(2012).Multidimensionaljournalevaluation:analyzingscientificperiodicals
beyondtheimpactfactor.Berlin:DeGruyter/Saur.Hicks,D.;Wouters,P.;Waltman,L.;DeRijcke,S.&Rafois,I.(2015).Bibliometrics:theLeiden
Manifestoforresearchmetrics.Nature520,429–431(23April2015)doi:10.1038/520429a.
Khodiyar,V.K.,Rowlett,K.A.andLawrence,R.N.(2014).Altmetricsasameansofassessingscholarlyoutput.LearnedPublishing,27:S25–S32.doi:10.1087/20140505
Kuhn,T.(1962).Thestructureofscientificrevolutions.Chicago:UniversityofChicagoPress.LaRivière,V.;Haustein,S.&Mongeon,P.(2015).Theoligopolyofacademicpublishers
19
inthedigitalera.PLOSONEJune15,2015.RetrievedJanuary15,2018fromhttps://doi.org/10.1371/journal.pone.0127502
Mabe,M.(2003).Thegrowthandnumberofjournals.Serials,16(2),191-197.Mabe,M.,&Amin,M.(2001).Growthdynamicsofscholarlyandscientificjournals.
Scientometrics,51(1),147-162.Mahood,S.(2015).Privatizedknowledgeandthepharmaceuticalindustry.In:Elliott,P.&
Hepting,D.FreeKnowledge:confrontingthecommodificationofhumandiscovery,pp.26-38.Regina:UniversityofReginaPress.
Mignolo, Walter D. (2002). The Geopolitics of Knowledge and the Colonial Difference.
South Atlantic Quarterly, 101(1), 57-96. Morrison,H.(2005).Theimplicationsofusagestatisticsasaneconomicfactorinscholarly
communications.In:Usagestatisticsofe-resources,D.Fowler(ed.).HaworthPress.Morrison,H.(2012).Thecommercializationandrationalizationofscholarlypublication.In:
Morrison,H.Freedomforscholarshipintheinternetage.Doctoraldissertation,SimonFraserUniversity.RetrievedJan.15,2018fromhttp://summit.sfu.ca/item/12537
Morrison,H.;Brutus,W.;Dumais-Desrosier,M.;Laprade,K.;Merhi,S.;Ouerghi,A.;Salhab,J.;
Volkanova,V.;Wheatley,S.(2017),Openaccessarticleprocessingcharges2016,doi:10.5683/SP/KC2NBV,ScholarsPortalDataverse.
Morrison,H.(2017).Elsevierasanopenaccesspublisher.TheCharlestonAdvisor18:3,pp.
53–59.RetrievedJanuary15,2018fromhttp://www.ingentaconnect.com/content/charleston/chadv/2017/00000018/00000003/art00014
Morrison,H.;Brutus,W.;Dumais-Desrosiers,M.;Kakou,T.L.;Laprade,K.;Merhi,S.;Ouerghi,
A.;Salhab,J.;Volkanova,V.;Wheatley,S.(2017).OpenAccessArticleProcessingCharges(OAAPC)LongitudinalStudy2016Dataset.Data2:13.
Odlyzko,A.M.(1994).Tragiclossorgoodriddance?theimpendingdemiseofscholarly
journals.JournalofUniversalComputerScience,0(0),1-52. Price,D.J.d.S.(1963).Littlescience,bigscience.NewYork:ColumbiaUniversityPress.
QSWorldUniversityRankings(2018).Methodology.RetrievedJanuary22,2018fromhttps://www.topuniversities.com/qs-world-university-rankings/methodology
RetractionWatch(n.d.).Top10mosthighlycitedretractedpapers.Website.RetrievedJanuary22,2018fromhttp://retractionwatch.com/the-retraction-watch-leaderboard/top-10-most-highly-cited-retracted-papers/
SanFranciscoDeclarationonResearchAssessment(2012).RetrievedJanuary15,2018
fromhttp://www.ascb.org/dora/
20
Schafer,A.(2015).Pseudo-evidence-basedmedicine:whenbiomedicalresearchbecomesanadjunctofpharmaceuticalmarketing.In:Elliott,P.&Hepting,D.FreeKnowledge:confrontingthecommodificationofhumandiscovery,pp.39-55.Regina:UniversityofReginaPress.
Schiltz,M.(2018).WhyPlanS?RetrievedFeb.18,2019fromhttps://www.coalition-s.org/why-plan-s/
ScholarlyPublishingandAcademicResourcesCoalition(SPARC)(n.d.).Bigdealcancellation
tracking.Website.RetrievedJanuary15,2018fromhttps://sparcopen.org/our-work/big-deal-cancellation-tracking/
ScienceEurope(2017a).ScienceEuropepositionstatement:onanewvisionformore
meaningfulresearchassessment.RetrievedJanuary15,2018fromhttp://www.scienceeurope.org/tallinn-call-for-action-takes-up-science-europe-recommendation-on-research-impact/
ScienceEurope(2017g).TaillinnCallforAction.RetrievedJanuary17,2018fromhttp://www.scienceeurope.org/tallinn-call-for-action-takes-up-science-europe-recommendation-on-research-impact/
Smith,R.(2006).Thetroublewithmedicaljournals.London:RoyalSocietyofMedicine
Press.Stephan,P.;Veugelers,R.&Wang,J.(2017).Reviewersareblinkeredbybibliometrics.
Nature544,p.411-412(27April2017)doi:10.1038/544411a.TimesHigherEducation(2018).WorldUniversityRankings2018.Inpartnershipwith
Elsevier.RetrievedJan.22,2018fromhttps://www.timeshighereducation.com/world-university-rankings/2018/world-ranking
U.K.OfficeofFairTrading.(2002).Themarketforscientific,medicalandtechnicaljournals.No.OFT396U.K.OfficeofFairTrade.RetrievedSeptember13,2011fromhttp://www.oft.gov.uk/advice_and_resources/publications/reports/media/
UniversityofOttawa&GovernmentofOntarioMinistryofAdvancedEducationandSkills
Development(2017).StrategicMandateAgreement2017-2020.Ware,M.&Mabe,M.(2015).TheSTMreport:anoverviewofscientificandscholarly
journalpublishing.4thed.TheInternationalAssociationofMedical,TechnicalandScientificPublishers(STM).RetrievedMay2,2017fromhttp://www.stm-assoc.org/2015_02_20_STM_Report_2015.pdf
21
AppendixTheAssociationofUniversityProfessorsoftheUniversityofOttawa(APUO)CollectiveAgreement(2016-2018)section23.3(assessmentofscholarlyactivities)states:
“23.3.1(a)Themembermaysubmitforassessmentarticles,booksorcontributionstobooks,thetextofpresentationsatconferences,reports,portionsofworksinprogress,and,inthecaseofliteraryorartisticcreation,originalworksandformsofexpression.(h)Itisunderstoodthatsincemethodsofdisseminationmayvaryamongdisciplinesandindividuals,disseminationshallnotbelimitedtopublicationinrefereedjournalsoranyparticularformormethods”.
Typically,anewfacultymemberishiredattherankofAssistantProfessorand,aftersixyears,appliesfortheseniorrankofAssociateProfessor,apromotionthatautomaticallyinvokestenure.ThecriteriaforresearchassessmentatthisstageiscoveredundertheAPUOCollectiveAgreementSection25.3.2.2(c)andreadsasfollows:
“(c)TheMemberhasproducedscientific,literary,artistic,orprofessionalworks--oracombinationthereof--whichare,inaccordancewiththecriteriasetforthin23.3.3.2,deemedofgoodquality.ThisassessmentshallbemadefollowinganoverallevaluationoftheMember'sscholarlyworks,carriedoutinaccordancewiththeprovisionsofsection23.3,duringwhichtheopinionofthree(3)outsideevaluatorswillhavebeenobtained,inaccordancewith23.3.2”.
23.3.3Levelofperformanceofscholarlyactivitiesstates:
*23.3.3.1WheneverthisagreementreferstosatisfactoryperformanceofscholarlyactivitiesbyaFacultyMember,itreferstoasituationwheretheMemberisregularlyengagedinscholarlyactivitiestheresultsofwhichindicatethatherperformance,incomparisontoarelevantgroupofpeersofcomparablerankandexperience,issatisfactory.*23.3.3.2TheMember'sscholarlyworksshallbeconsideredgoodiftheyrepresentacontributioninadditiontothatcontainedintheMember'sdoctoralthesisortotheworkthathasbeentakentobetheequivalentofadoctorate,andif,subsequenttothatwork:(a)inthecaseofresearch,theydemonstratecontinuousprogressinthedevelopmentoftheMember'sresearchactivitiesandcontributetotheadvancementofknowledgeintheMember'sfieldofspecialization;(b)inthecaseofliteraryorartisticworks,theyattesttocontinuouscreativeactivity,well-reputedintheliteraryorartisticcommunityoutsidetheUniversityofOttawa;(c)inthecaseofprofessionalworks,theyattesttothepracticeofaprofessionaboveandbeyondthatwhichisgenerallyexpectedofanon-teaching,practicingprofessional,ortheycanbeconsideredasavaluablecontributiontotheadvancementoftheprofessionitself.
Recommended