1

Whatcountsinresearch?Dysfunctioninknowledgecreation&movingbeyondbyHeatherMorrisonBiostatement:HeatherMorrisonisanAssociateProfessorintheSchoolofInformationStudies,andcross-appointedtotheDepartmentofCommunication,UniversityofOttawa.Herresearchfocusistheeconomicsoftransitionofscholarlypublishingfrompurchaseandsubscriptionstosupportforproductiontofacilitateopenaccess.HeatheristhePrincipleInvestigatoroftheSSHRCInsightGrant(2016-2021)supportedSustainingtheKnowledgeCommonsresearchprogram.AbstractThischapterbeginswithabriefhistoryofscholarlyjournalsandtheoriginsofbibliometricsandanoverviewofhowmetricsfeedintouniversityrankings.Journalimpactfactor(IF),ameasureofaveragecitationstoarticlesinaparticularjournal,wasthesoleuniversalstandardforassessingqualityofjournalsandarticlesuntilquiterecently.IFhasbeenwidelycritiqued;evenClarivateAnalytics,thepublisheroftheJournalCitationReports/IF,cautionsagainstuseofIFforresearchassessment.Inthepastfewyearstherehavebeenseveralmajorcallsforchangeinresearchassessment:the2012SanFranciscoDeclarationonResearchAssessment(DORA),the2015LeidenManifesto(translatedinto18languages)andthe2017ScienceEuropeNewvisionformeaningfulresearchassessment.Meanwhile,duetorapidchangeintheunderlyingtechnology,practiceischangingfarmorerapidlythanmostofusrealize.IFhasalreadylargelybeenreplacedbyitem-levelcitationdatafromElsevier’sScopusinuniversityrankings.Altmetricsillustratingawiderangeofusesincludingbutmovingbeyondcitationdata,suchasdownloadsandsocialmediauseareprominentlydisplayedonpublishers’websites.Thepurposeofthischapteristoprovideanoverviewofhowthesemetricsworkatpresent,tomovebeyondtechnicalcritique(reliabilityandvalidityofmetrics)tointroducemajorflawsinthelogicbehindmetrics-basedassessmentofresearch,andtocallforevenmoreradicalthoughtandchangetowardsamorequalitativeapproachtoassessment.ThecollectiveagreementoftheUniversityofOttawaispresentedasonemodelforchange.Abriefhistoryofjournals,bibliometrics&rankingsIn1665,twoscholarlyentrepreneursindependentlyseizedthepotentialoftheprintingpressandthepostalsystem,andinventedthemodernscholarlyjournal.Guédon’s(2001)InOldenburg’slongshadowpresentsanoverviewofthehistoryofthescholarlypeer-reviewedjournalfromitsinceptionin1665withOldenburg’sPhilosophicalTransactionsanddeSallo’sJournaldesSçavans,totheendofthe20thcentury.Theideaofpeerreviewhasevolvedovertime,buttheformatofjournalshasremainedlargelythesame.Odlyzko(1994)predictedtheimpendingdemiseofscholarlyjournals.Printandmailareintheprocessofbecomingobsoleteasthestandardforproductionanddisseminationofscholarlywork,asthisbecomeselectronicandweb-based.Thecontinuityoftheprint-basedformat,withonlinejournalscloselyresemblingprintones,reflectsacceptanceofthescholarlyjournalarticleasthegoldstandardforpublicationinmanyacademicdisciplines.Thegrowthofscholarlyjournalsandarticlessince1665hasbeenremarkablyconstant.ThiswasfirstdocumentedbyDeSollaPrice(1963,p.17)inLittleScience,BigScience,andupdatedbyMabe&Amin(2001)andMabe(2003).Thereisanaverageannualscholarly

2

journalandarticlegrowthrateofabout3-3.5%peryearfromthe1600stothepresentday.Iftherewerestilljusttwoscholarlyjournalsproducingasmallvolumeofarticlesonanannualbasis,itwouldbefeasibleforeveryscholartoreadeveryscholarlyarticle.However,asthevolumeofproductiongrew,journalsbegantospecializeinparticulardisciplinesandsub-disciplines,atratesvaryingwiththegrowthofthedisciplines.Asproductioncontinuedtoincrease,specializationwasnotenough.Guédon(2001)arguesthatthegrowingnumbersofjournalswastheinspirationforatendencytowanttodefine“corejournals”.Thepurposeofthe“corejournal”conceptwastoaddresstwoproblemsthataroseasthenumberofjournalsgrew.Oneproblemwasthe“serialscrisis”documentedbytheAssociationofResearchLibraries(1989),acombinationofincreasingnumbersofjournalsandaveragepricerisesforjournalsbeyondinflation,yearafteryear,leadinglibrariestocancelsubscriptions.Asecondproblemwastheincreasingdifficultyscholarshadinkeepingupwiththegrowingliterature.Inthatsense,identifying“corejournals”wouldhelpbusyscholarsprioritizetheirreadingsandpublicationvenues.Garfield(1955)proposed“abibliographicsystemforscienceliteraturethatcaneliminatetheuncriticalcitationoffraudulent,incomplete,orobsoletedatabymakingitpossiblefortheconscientiousscholartobeawareofcriticismsofearlierpapers”.Anotherproposedpurposeofthissystemwastofacilitatecommunicationamongscientists.ItwasinthisarticlethatGarfieldfirstcoinedtheterm“impactfactor”(IF),athen-hypotheticalmeasureoftheinfluenceofahighlycitedarticle.Garfield(2006)describesthehistoryofthedevelopmentofcitationindexingandIF.WithsupportfromtheU.S.NationalInstitutesofHealth,IFbecamethebasisforthedevelopmentoffirsttheGeneticsCitationIndexandlatertheScienceCitationIndex.IFisametricappliedtojournalsratherthanarticles.Itisbasedon2elements,anumeratorconsistingofthenumberofcitationsinthecurrentyeartoitemspublishedinaparticularjournalintheprevious2years,andthedenominator,thenumberofsubstantivereviewsandarticlespublishedinthesame2years.Inotherwords,IFistheaveragenumberofcitationstoanarticleinaparticularjournalfortheprevioustwoyears.IFvariesconsiderablybydisciplineandsub-disciplineaswellasbyjournal,andisoftenevaluatedonthebasisofthestatusbyquartilewithinadiscipline.TheScienceCitationIndexdevelopedbyGarfieldandcolleaguesin1961hasmorphedandgrownintoWebofScience,includingtheScienceCitationIndex,theSocialScienceCitationIndex,theArts&HumanitiesCitationIndex,theEmergingSourcesCitationIndex,theBookCitationIndex,theConferenceProceedingsCitationIndex,inadditiontooptionalspecializedcollections.WebofScienceisoneofasuiteofinter-relatedproductsproducedandsoldbyClarivateAnalytics(untilrecentlypublishedbyThomsonReuters),andthebasisfortheirJournalCitationReports(JCR)whichprovidesreportsofjournalIF.

3

Table1:coreClarivateproductsTable1illustratestherelationshipandevolutionofthecoreproductsrelatedtoresearchandresearchmetricsofferedbyClarivateanalytics.Theinitialcoreunderlyingproductisamassivedatabaseofcitationstojournalarticles,WebofScience.TheresearchdiscoverytoolcalledWebofScienceisextensivelyusedbyresearchersatuniversityandresearchlibrariesforresearchdiscovery.ThesameunderlyingmetadataisusedforthetraditionalJournalCitationReports(JCR).ThisisthetooloriginallyenvisionedbyGarfieldasameansofidentifyingasetof“core”ormosthighlycitedjournalssothatresearcherscouldprioritizetheseforreadingandlibrariesforpurchase.JCRisimportantasabrandingtoolforpublishers.JCRisalsousedbyuniversityrankingsagencies.Ontheright,morerecenttoolsInCitesandEssentialScienceIndicatorsfocusexclusivelyonmetricsforevaluation.Thesearetoolsformeasuringresearchers,notassistingresearchersintheirwork.Atasurfacelevel,informationondifferentpagesontheClarivatewebsitemightappearconfusingandcontradictory.ThisisbecauseClarivateoffersservicestoadiversegroupofstakeholdersthathavedifferentgoalsthatarenotalwayscompatible.OntheJCRwebsite,ClarivateclaimsthatJCR“givesyouasystematic,objectivemeanstoevaluatetheworld’sleadingscientificandscholarlyjournals.Byanalysingcitationreferences…JCRmeasuresresearchinfluenceandimpactatthejournalandcategorylevels,andshowstherelationshipbetweencitingandcitedjournals”(ClarivateAnalytics,n.d.).JCRincludes11,000journalsfromover230disciplines.Thisisaboutathirdofthetotalactivepeer-reviewedjournalsreportedbyWare&Mabe(2015)inarecentstateoftheindustryoverviewreportproducedfortheInternationalAssociationofScientific,TechnicalandMedicalPublishers(STM)(28,100intheEnglish-language,6,450inotherlanguagesthanEnglish).ThetargetmarketsforJCR,accordingtotheClarivatewebsiteasofOctober16,2018arelibrarians,toinformpurchaseandcancellationdecisions;publishersandeditors,toassesstheeffectivenessofjournalsinthemarketplace;researchers,toidentifythemostinfluentialjournalsinwhichtopublish,andresearchmanagersandinformationanalyststo“track

WebofScience(Citationmetadata)

ResearchDiscovery

WebofScience:universityandresearch

libraries

ResearchEvaluation

InCitesResearchorganizations,

funders

JounalCitationReports:universityandresearchlibraries,publishers,universityrankings

agencies

EssentialScienceIndicators

Researchorganizations,fundingagencies,

publishers

4

publicationandcitationpatternstoaidyourstrategyandpolicydecisions”.ClarivateAnalytic’s(n.d.)webadviceonthesuitabilityoftheuseofJCRandotherdataderivedfromWebofScienceinresearchassessmentisdependentonthetargetaudienceoftheirdifferentproducts.ThereisamarkedcontrastbetweenadviceontheJCRwebsite(librariansastheprimarytargetaudience)andInCites(researchorganizationsandfundingandpolicyorganizationsasthetwoprimarytargetaudiences).OntheJCRwebsite,ClarivatewarnsagainsttheuseofIFinassessingjournalquality,statingthat“ClarivateAnalyticsdoesnotdependontheimpactfactoraloneinassessingtheusefulnessofajournal,andneithershouldanyoneelse…Theimpactfactorshouldbeusedwithinformedpeerreview.Inthecaseofacademicevaluationfortenureitissometimesinappropriatetousetheimpactofthesourcejournaltoestimatetheexpectedfrequencyofarecentlypublishedarticle.Again,theimpactfactorshouldbeusedwithinformedpeerreview.Citationfrequenciesforindividualarticlesarequitevaried.”Incontrast,thetitleoftheClarivatewebsiteforInCites(likeJCR,basedonWebofSciencedata)statesthatitis“anobjectiveanalysisofpeople,programsandpeers[emphasisadded]”.RecommendeduseofInCitesdataforassessingindividualresearchersasaprimaryuseisimpliedforeachtargetaudienceintheir“Who’sitfor”.Forresearchorganizations,InCitesispresentedasameansto“identifyandmanageresearchactivitiesandtheirimpact”aswellasto“identifyexperts”.Forfundingorganizations,usessuggestedinclude“identifyemerging…researchersandexperts”and“managefundingactivityfromsubmissionstoprogressreportsthroughoutcomes”.ThisassumesthatemergingresearchersandexpertscanbeobjectivelyidentifiedthroughWebofSciencedata,i.e.researcherswhopublishinhighIFjournalsarepresumably“emerging”and“expert”.ItisalsoassumedthatpublicationinhighIFjournalsandhighcitationcountsareobjectivemeasuresofthequalityofresearch.PublishersaretoldthatInCitesprovidesameansto“identifythebestauthorsandreviewers”.WareandMabe(2015)discussincreasingindustrycriticismoftheuseofcitationdata,particularlyIF,tojudgethequalityofindividualresearchersanddepartments.Willthisindustryrecognitionleadtochange,andifsowhatformwillthischangetake?Basedonmajoruniversityrankingagencies’descriptionsoftheirmethodsandElsevier’sdescriptionofScopus(n.d.),itappearsthatanevolutionfromjournal-basedmetrics(IF)tobibliometricsbasedonindividualworks(articles,booksandbookchapters,conferenceproceedings)hasalreadytakenplaceinalargesectorofthemarket.ThisshiftaddressesamajortechnicalcritiqueofIF,usingjournalIFasasurrogateforarticleimpact.Elsevier’sScopusisthemajorsourceofdatafortheTimesHigherEducation(THE)andQSWorldUniversityRankings,amongothers.Assessmentofresearchandresearchersisoftenconductedbyresearchersthemselves,forexampleinpromotionandtenuredecisionsandinreviewofgrantapplications.Whileuniversityrankingssystemsaremovingtowardsarticle-levelmetrics,researchers’ownpracticesaredeeplyingrainedinacademiccultureandcontinuetorelyprimarilyonIF.Stephan,Veugelers&Wang(2017)discusswhattheycall“back-doorbibliometrics”,inwhichresearchersandreviewersreportand/orusejournalIFinassessmentevenwhenthisisnotrequired.Theyalsodiscusstheformaluseofbibliometricindicators,suchastheuseofrankingsderivedfromjournalIFinSpaininpromotionandsalaryincreasedecisions,andpaymentofbonusesinChinaaccordingtotheprestigeofthejournalinwhicha

5

researcherispublished.InsomeregionssuchasFlandersandBraziljournalIFisusedinallocatingresourcestouniversities.Whenresearchersfocusontheirownareasofspecialization,onemightassumethattheyhavethebackgroundknowledgetounderstandcommonlyusedmetrics.However,relianceonmeasuresandsurrogatemeasuresofjournalandarticleinfluenceiscommon,althoughresearchmetricsperseisnotacommonresearchspeciality.Tofullyunderstanduniversityrankings,weneedtoknowwhoproducesthedatathatfeedsintotherankings,andhowtheyproduceit.Elsevier’sScopusdataisthebasisfor38.5%oftherankingforTHE’sWorldUniversityRankings(WUR)(THE2018,p.82-83).Citationsorresearchinfluenceaccountfor30%oftheTHEWUR.Thesearebasedon“almost62millioncitationstomorethan12.4millionjournalarticles,articlereviews,conferenceproceedingsandbookandbookchapterspublishedover5years”.AccordingtoTHE,thesedata“helptoshowushowmucheachuniversityiscontributingtothesumofhumanknowledge…whoseresearchhasstoodout…beenpickedupandbuiltonbyotherscholars…”(p.83).THE’s“ResearchProductivity”isacountofthe“numberofpaperspublishedintheacademicjournalsindexedbyElsevier’sScopusperscholar,scaledforinstitutionalsizeandnormalisedforsubject”.Under“InstitutionalOutlook”,“InternationalCollaboration”isameasureofportionoftheuniversity’stotalresearchjournalpublicationsthathaveatleastoneinternationalco-author.ThetitlepaperoftheTHEreportstatesthattheworkis“inpartnershipwithElsevier”.QS’(2018)WorldUniversityRankingsusesCitationsperFaculty,astraightforwardcountofcitationstotheworksofscholarsattheuniversitybeingevaluated,usingtheScopusdatabase.Maclean’s(Dwyer,2017)addedbibliometricsindicatorsin2015,publicationsperfacultyandafield-weightedcitationimpactfactor,drawnfromScopus.Neworalternativemetrics-basedapproachesArticlemetricsisatpresentinaprocessofrapidevolution.Severalbasictrendsareobservable:ashiftfromjournaltoarticlelevelcitationmetrics,discussedabove;newtypesofmetricsoraltmetricsthatillustratedifferenttypesofindicatorsofusage,suchasviews,downloads,andsocialmediausage;andinclusionofmetricsandlinkstodownstreamcitingarticles,socialmedia,etc.,onpublisherwebsites.Inthissection,Iwillpresentabriefoverviewillustratingtherapidimplementationofdiverseapproaches,explaininplaintermswhatresearchersandpublishersinthisareaareaimingtoaccomplish,andarguethatwhilesomeaspectsofthesedevelopmentsareusefulforresearch,thereisaproblematiclackofcriticalreflectionontheimpactofthesedevelopments.In2012,Hausteinpublishedacomprehensivebookonthetechnicaldetailsandflawsofscholarlybibliometricsasofthatpointintime,concludingwitharecommendationforamultidimensionalapproachtometricstoovercometheflawsevidentinanyonemethod.Manyotherauthors,suchasKhodiyar,Rowlett,&Lawrence(2014),havesimilarlydiscussedthechangingnatureofassessmentofscholarlywork.Atpresent,thestateofpracticehasfaroutpacedscholarlyconceptionsofnewapproaches.Asdiscussedintheprevioussection,whileresearcherscontinuetoassumethatthejournalIFisstateoftheartinmetrics-basedevaluation,majoruniversityrankingsandtheworld’s

6

largestcommercialscholarlyjournalpublisher,Elsevier,havealreadymovedtoarticle-levelcitationmetricsusingdatafromtheirproductScopus.Meanwhile,publishingpracticealreadyreflectsheavyuseofneworaltmetricsthatincludebeyondcitations.Thestateofpracticecanbeeasilyobservedbybrowsingthewebsitesofscholarlyjournals.

Figure1.Elsevier’sJournalofDevelopmentEconomics.Source:https://www.journals.elsevier.com/journal-of-development-economicsOntheright-handsideofthehomepageofElsevier’sJournalofEconomicDevelopmentthereisameansforreaderstofilterarticlesthatincludestheoptions“mostdownloaded”and“mostcited”.Ontheleft-handsideofthepage,asillustratedinFigure1,thereisalistof5journalmetricsandalinkto“viewmoreonjournalinsights”;thispageincludesevenmoremetrics.Eachmetrichasanicon“I”formoreinformation;hoveringovertheiconbringsforwardthetechnicalexplanationforeachmetric.

Figure2:Trending.Source:NatureScientificReports.https://www.nature.com/srep/IfyouscrolldownthehomepageofNature’sopenaccessjournalScientificReportsyouwillseeasectioncalled“Trending”withtheword“Altmetric”prominentlydisplayedatthetoplefthandcornerasillustratedinFigure2.

7

Figure3.Articlemetricsfordinosaurarticle.Source:https://www.nature.com/articles/s41598-018-37943-3/metrics

Figure4.MentionsandTwitterdemographicsfordinosaurarticle.Source:https://www.nature.com/articles/s41598-018-37943-3/metricsClickingonthetoptrendingarticlebyaltmetric(dinosaurarticle)revealsmoredetailaboutthemetricsinvolved.AsofFebruary12,2019,thereare0citationsfromWebofScience.OnlineAttentionindicatestweeting,Facebook,andmediaattentionasillustratedinthefollowingFigures.Figure3showstotalcitations,onlineattentionandthealtmetricsscore.Figure4illustratesfurtherdetailthatcanbefoundbyscrollingdownfromFigure3,aclickablelistofmediareferencesandanoptiontoswitchtoscientificblogs,aswellasamapillustratingTwitterreferencesandalistoftweetingcountriesindescendingorderbynumberoftweets.Furtherdownonthepage(notillustratedhere)areexplanationsoftermsandsources.

8

TheseElsevierandNaturejournalseachreportneworaltmetrics,butnotthesameones.Why?Elsevierusescitationdatafromitsownproduct,Scopus.NaturePublishingGroupusescitationdatafromClarivate’sWebofScienceandCrossRefforScientificReportsratherthanScopusdata.Couldthisbebecausethetwocompaniesarerivals?NaturePublishingGroupownsSpringer,theworld’ssecond-largestcommercialscholarlypublisherandhenceamajorcompetitorforElsevier.PerhapsNatureprefersnottodisplayElsevier’sScopusdata,oritmightbethatElsevierprefersnottoprovideNaturewithareasonablepriceforuseofScopusdatatoacompetitor.PublicLibraryofScience(PLOS)providesadetailedexplanationoftheirarticle-levelmetrics(ALMs)onthePLOSwebsite:https://www.plos.org/article-level-metrics.PLOSdefinesALMSas:“quantifiablemeasuresthatdocumentthemanywaysinwhichbothscientistsandthegeneralpublicengagewithpublishedresearch”.SuggestedusesofALMsforresearchersaretocommunicateimpactingeneralandtofunders,toraisearesearcher’scareerprofile,andtofindcollaborators.

CallsforchangeinresearchassessmentDORAThefirstmajorcalltoactionisthe2012SanFranciscoDeclarationonResearchAssessment(DORA)initiatedbytheAmericanSocietyforCellBiology.DORA'srecommendationsstate"theneedtoeliminate[emphasisadded]theuseofjournal-basedmetricssuchasJournalImpactFactorsinfunding,appointment,andpromotionconsiderations".AsofOctober2018,DORAhasbeenendorsedbyover600organizationsand13,000individuals,myselfincluded.DORAdoesnotquestiontheconceptofmeasurementperse,statingthat:“fundingagencies,institutionsthatemployscientists,andscientiststhemselves,allhaveadesire,andneed,toassessthequalityandimpactofscientificoutputs.Itisthusimperativethatscientificoutputismeasuredaccuratelyandevaluatedwisely”.ImplicitinDORAisanassumptionthatthepeer-reviewedjournalarticlewillcontinuetobethemostfrequentmeansofdisseminationofnewknowledgeintheforeseeablefuture.Developmentofmetricstoincludenewformsofresearchoutputssuchasdatasetsandsoftwareisencouraged.DORAincludesgeneralrecommendationsandspecificrecommendationsforfundingagencies,institutions,publishers,organizationsthatsupplymetrics,andresearchers.Thelatter,wheninvolvedincommitteesmakingdecisionssuchashiring,promotion,andtenure,areencouragedtomakeassessmentbasedonscientificcontentratherthanpublicationmetrics.Oneofthedeficienciesofallcitation-basedmetricsnotedinDORAisaskewtowardsreviewarticlesasauthorstendtocitereviewarticlesratherthantheoriginalworksthatarereviewed.Publishersandresearchersarecalledupontoencourageauthorstociteoriginalresearch.Itisrecommendedthatfundingagenciesandinstitutions“considerabroadrangeofimpactmeasuresincludingqualitativeindicatorsofresearchimpact,suchasinfluenceonpolicyandpractice”.However,publishersareencouragedto“makeavailablearangeofarticle-levelmetrics”andresearchersto“usearangeofarticlemetricsandindicators”,whichsuggestsadeeperquantitativeratherthanaqualitativeapproach.Publishersareaskedto“encourageresponsibleauthorshippracticesandtheprovisionofinformationaboutthespecificcontributionsofeachauthor.”Qualitativeinformationabout

9

authorcontributionswouldappeartoadvancethepotentialforenhancedqualitativeassessment.LeidenManifestoAgroupofself-describedscientometricians,socialscientistsandresearchadministratorshighlightsomeofthemajorissueswithjournalIFandothermetricsandlist10principlestoguideresearchevaluation,principlesthatwerecrystallizedatthe19thAnnualConferenceonScienceandTechnologyConferenceIndicatorsheldinLeidenin2014;hencethislistisreferredtoastheLeidenManifesto(Hicksetal.,2015).TheLeidenManifestohttp://www.leidenmanifesto.org/hasbeentranslatedinto18languages.Thefirstprinciplestates:“quantitativeevaluationshouldsupportqualitative,expertassessment”.Theauthorscitean“impactfactorobsession”,stating“soaringinterestinonecrudemeasure–theaveragecitationcountsofitemspublishedinajournalinthepasttwoyears–illustratesthecrisisinresearchevaluation”.CritiqueofmetricsisnotlimitedtoIF.Simplepublicationcountscanbeproblematicaswell.AnexampleisprovidedofarelativelylowratingofagroupofEuropeanhistoriansinanationalpeer-reviewexercisesimplybecausehistorianstendtowritebooksratherthanjournalarticles.H-factorincreaseswiththeresearcher’sage,eveniftheresearcherdoesnotproducenewpapers.Theh-factorisalsodatabase-dependent.Computerscientistscanhaveanh-factorof10inWebofSciencebut20–30inGoogleScholar.Precisionmatters;“asinglehighlycitedpublicationslightlyimprovesthepositionofauniversityinarankingthatisbasedonpercentileindicators,butmaypropeltheuniversityfromthemiddletothetopofarankingbuiltoncitationaverages”Theauthorssuggestthatrelyingonasinglemeasurewill“invitegamingandgoaldis-placement(inwhichthemeasurementbecomesthegoal)”,andoneoftheirproposedsolutionsismultiplemetrics.LeidenPrinciple2,isto“Measureperformanceagainsttheresearchmissionsoftheinstitution,grouporresearcher”.Thereisnosinglemetricthatmakessenseineveryresearchcontext.Forexample,consideranactionresearchprojectdesignedtohelpacommunitygroupaddressanissueofconcerntothem.Ideally,designoftheproject’sgoalsandevaluationmeasuresshouldbeundertakeninconsultationwith,orledby,thecommunitygroup.Theoptimalmeasuresofsuccesswouldprobablybereal-worldindicatorsofchange.Homelesspeoplewanthomes,notcitationstoarticlesabouthomelessness.Publicationofresults,writteninplainlanguage,invenuesthatarephysicallyandintellectuallyaccessibletothecommunity,suchasthecommunity’sownnewsletterorblogoralocalworkshopmaybemoreeffectiveinmeetingthegoalsoftheresearchthanpublicationinscholarlyjournalsusingacademicjargonthatthegroupmaynothaveaccesstoorunderstand,orpresentationsatscholarlyconferencesthatgroupmemberscannotaffordtoattend.Thesameprincipleswouldapplytoacademic/industryandacademic/governmentcollaborations.Thisisnottosaythattraditionalacademicideasofexcellencedonotapply,ratherthatmeasuringexcellencebynumberofpublicationsandcitationsinprestigiousjournalsisnottheoptimalwaytoevaluateeverytypeofresearchproject.LeidenPrinciple3states:“protectexcellenceinlocallyrelevantresearch”.OneexampleoftheproblemisthatofSpanishlawwhichstatesthedesirabilityofSpanishscholarspublishinginhigh-impactjournals.Insociology,thehighestimpactfactorjournalsarepublishedinEnglishintheUnitedStates(US);likelyasaresultofthis,highlycitedSpanishsociologistsarethosewhofocusoneitherabstractmodelsorUSsocialproblems.

10

ScienceEurope’sNewvisionformoremeaningfulresearchassessmentInJuly2017,ScienceEurope(2017a),anon-profitorganizationbasedinBrusselsrepresentingmajorresearchorganizationsacrossEurope,issuedapositionstatement“onanewvisionformoremeaningfulresearchassessment”.Thepreamblecontraststhebroadimpactofresearchonsociety,oftengainedthroughagradualdevelopmentofnewknowledge,withmetricsdesignedtomeasuretheimpactofaspecificstudy,andpointsoutthatitisnotalwayspossibletoconnectsocietalimpactwithaparticularresearchstudy.Thepreamblegoesontodiscusstheconceptofthevalueofresearch,abroadernotionofresearchimpactthatincludessocietalvalues.Societalprogressdrawsfrombothresearchoutputsandothersources;toonarrowanemphasisonconcreteimpactmaygenerateunintended,andnotnecessarilybeneficial,effectsonresearchactivity.Theconclusionstates“thereisgreatdiversityinthewaysinwhichresearchbringsitsimmensevaluetosociety.Someofthesewaysareindirectorintangibleandcannoteasilybemeasuredbystrictlydefinedimpactassessmentcriteria.Othersarelong-termorunpredictableandmaynotyetbevisibleatthetimethattheresearchisevaluated...ultimately,thebestwaytomaximisethevalueofresearchtosocietyisbyensuringthattheresearchproducedmeetsthehigheststandardsofqualityandexcellence”.ScienceEurope’sstatementemphasizestheimportanceoftrustbetweenresearchersandsociety.ThisisreflectedinPriority2ofScienceEurope’s(2017b)TaillinnCallforAction,whichcallsforresearchorganizationsandfundersto“recogniseabroadnotionofimpactthatacknowledgesthesocietalvalueofresearchforpolicyandpractice”andcallsonpolicymakers,researchfunders,andacademicto“fosterthenecessaryculturalchangetoembracethebroadnotionofimpact”.Implementationofchangeinapproachestoresearchevaluationisinanearlystage.InSeptember2018theEuropeanCommissionendorsedPlanSandCoalitionS,anambitiousplantoacceleratethetransitiontoopenaccesspublishing;“thewayweevaluateresearchoutputs”isidentifiedasoneofthebarrierstochange(EuropeanCommission,2018).ScienceEurope’sPresidentMarcSchiltz(2018)inastatementcalledWhyPlanS,statesthat“Wecommittofundamentallyrevisetheincentiveandrewardsystemofscience,usingtheSanFranciscoDeclarationonResearchAssessment(DORA)asastartingpoint”. The basic idea is for all funding agencies, particularly Europe but the aspirations are to inspire change globally, to commit to changing how research is evaluated, from traditional to new or altmetrics. It will be interesting to observe progress towards implementation of this ambitious plan over the next few years.

DiscussionMetricsversusqualityinresearchCanmetricsystemscapturetheessentiallyqualitativenatureoftheconceptofqualityinresearch?ThemaingoaldrivingdevelopmentofcitationindexingasdescribedbyGarfieldin1955wassothatresearcherscouldtrackforwardsfrompublishedworktocitingworksthatmightpointoutcritiqueoftheoriginal.Iarguethatthewholeideaofusingmetricstoassessthequalityofresearchandresearchersisrelativelynewandhasnotreceivedthecriticalattentionthatitdeserves.Thissectionaimstobeginaprocessofapplyingscholarly

11

critiquetothisarea.AsDORAnotes,thereisaskewincitationstowardsreviewarticlesratherthanoriginalresearch.Thisraisesaquestion:ifresearchersarecitingreviewarticlesratherthanoriginalresearch,aretheyevenreadingtheoriginalresearch,nevermindtrackingdownstreamcitations?Ifweassumethatreviewarticlescontainalltheimportantinformationfromeveryreviewedarticle,captureitaccurately,andthattheoriginalarticlesarenevercritiqued,retractedorrefutedbysubsequentresearch,thenreadingreviewarticlesisnotproblematic,butthenwouldbenoneedforcitationlinkingtouncovercritique.Currentapproachestoresearchassessmentassumethatwhenitcomestoscholarlypublishing,moreisbetter.Giventheconstantincreaseinthevolumeofproductionofscholarlyworksandtheavailabilityofcitationindexingtopermitmorecarefulchecking,shouldn’tresearcherstodaybespendingrelativelymoretimereadingratherthanpublishing?Iftheywere,wouldn’ttheybepublishinglessratherthanmore?Assessmentsystemsbasedonthepremisethatmoreisbetterseemlikelytoriskincreasingerrorssuchasinvalidresults.Researchintocurrentpracticewouldbehelpful.Onemightsurveyresearchersonwhethertheyactuallyreadalloftheworksthattheycite,whethertheyrelyonsecondarysourcessuchasreviewsorgototheoriginals,andwhethertheyusecitationindexestocheckdownstreamcitingsources.Or,onemightanalysewrittenpublicationstoseewhetherthereareerrorsthatmighthavebeencaughtwithmorein-depthreading.Advancingourknowledgerequiresquestioningunderlyingassumptionsinadditiontobuildingonexistingwork.Twoassumptionsintheareaofresearchassessmentthatshouldbechallengedaretheassumptionthat“impact”itselfisnecessarilypositivelycorrelatedwithgoodqualityworkandthatimpactisinherentlydesirable.ThesecondmosthighlycitedretractedpaperaccordingtotheRetractionWatch(n.d.)blogistheinfamous1998paperbyWakefieldetal.publishedinthehighlyprestigiousjournalLancet,purportingtomakeaconnectionbetweenvaccinationandautism.Thisarticlehasbeencitedover1,000timesinthelistofjournalsincludedinWebofScience,with640citationsbeforeretractionand468citationsafter.Anyoftheexistingoremergingmetrics-basedapproachestoresearchassessmentswouldfindthatthisstudyhashadalotofimpact.ThearticlewaspublishedinahighIFjournal.Itisahighlycitedarticle,whichwouldresultinhigharticle-levelrankingsandboosttheh-factorofalloftheauthors.Ifweconsiderreal-worldimpact,theinfluenceofthisarticleintheanti-vaccinationmovementandthesubsequentreturnofdiseasessuchasmeaslesdemonstrateanexceptionalreal-worldimpactforasinglearticle.Thisillustratesthedangerofassumingthatimpactisnecessarilygood.Likealmostallqualitiesofthingsintherealworld,impactisneithergoodnorbadinandofitself,butrathermustbeinterpretedincontext.Ifuniversitiesandresearchfundersarerelyingonuniversityrankings,thisprovidesanadditionalincentivetofocusontraditionalformsofpublication.Forexample,thebibliometricspartnerofTimesHigherEducation(2018)inproducingtheWorldUniversityRankingsisElsevier.Tomeasureresearchproductivity,Elseviercounts“thenumberofpaperspublishedintheacademicjournalsindexedbyElsevier’sScopusdatabaseperscholar”.Tocalculatecitationsorresearchinfluence,Elsevier“examined…citationstojournalarticles,articlereviews,conferenceproceedingsandbooksandbookchapters”.Thiscountofcitationstoparticulartypesofworksisadefactoendorsementofthesetypesofworks.ItisprobablynotacoincidencethatElsevierisahighlyprofitablepublisherofexactlythesetypesofworks.

12

Myownexperienceasascholarconfirmsthisfocusonalimitedrangeofformats.Ouruniversity’sonlineCVsystemislikelytypicalincategorizingtypesofpublications–booksandbookchapters,peer-reviewedjournalarticles,non-peer-reviewedarticlesandsoforth.Thiscategorizationisunderstandableforhistoricalreasons,howevertheendresultisthatthemajorityofmyworks,andalmostallthatIconsidermymostimportantandleading-edgeworks,suchastheopendatadiscussedaboveandmyscholarlyblogs,SustainingtheKnowledgeCommonsandTheImaginaryJournalofPoeticEconomicsarelabelledas“other”andwouldcountforlittleornothingunderexistingmetrics-basedassessmentapproaches.RetractionWatchhttp://retractionwatch.com/bloggerstrackandreportonpublishedarticlesthatwereretractedafterpublication.UnlikeJCR,InCites,orScopus,RetractionWatchaddressestheoriginalmaingoalofcitationindexesasproposedbyGarfieldin1955,“abibliographicsystemforscienceliteraturethatcaneliminatetheuncriticalcitationoffraudulent,incomplete,orobsoletedatabymakingitpossiblefortheconscientiousscholartobeawareofcriticismsofearlierpapers”.Mostretractionsreflecterrorsdiscoveredafterpublicationbutalsooccasionallyfraudisuncoveredandreported.RetractionWatch’s(n.d.)top10mosthighlycitedretractedpaperslists10papersthathavebeencitedmorethan550timesinjournalsindexedinWebofScience(asofJanuary2018),andthislistincludes2papersthathavebeencitedmoreoftenafterretractionthanbefore.Theblogalsotracksevidencethatcitationofretractedpapersisanongoingproblem.RetractionWatchdemonstratesafundamentalflawwithcurrentapproachestoresearchassessment,whichfocusesontheimpactofscholarlywork,whethermeasuredindirectlythroughmetricssuchasjournalIFordirectlythrougharticle-levelmetrics,andforthemostpartneglectsthemoreimportantquestionoftheaccuracyofscholarlywork.BibliometricsandtheeconomicsustainabilityofscholarlycommunicationIncreasingmarketconcentrationwasthesubjectofinvestigationbytheUnitedKingdom(UK)OfficeofFairTrading(2002).AsdescribedbyMorrison(2012),industryconcentrationandgrowingprofitsofafewlargecommercialscholarlyjournalpublisherswasaccompaniedbyasignificantdecreaseintheaveragenumberofcopiesofscholarlymonographsproducedandsold.Recently,LaRivière,Haustein&Mongeon(2015)reportanincreaseinconcentrationinthescholarlypublishingmarket,withthetop5publishersaccountingformorethan50%ofthearticlesindexedinWebofScience.TheScholarlyPublishingandAcademicResourcesCoalition(SPARC)(n.d.)maintainsalistofjournalbigdealcancellationsbyuniversitylibraries,librarygroupsandstatelibrarysystems,andnationalcoalitionssuchastheConsortiumonCoreElectronicResourcesinTaiwan(CONCERT).Thisisnotahealthysystem;thehighpricesandprofitsofafewcommercialscholarlypublisherscannotbesustainedbyacademiclibraries,andtheeconomiccloutbehindthebigpackagedealsresultinlittlefundingleftoverforpublishingscholarlymonographsandsupportingthejournalsofsmallerpublishers,particularlyinthehumanitiesandsocialsciences.Myresearchsuggeststhesametrendofcommercialconcentration,involvingthesamecompanies,isemerginginopenaccesspublishing.Asof2017,thelargestopenaccessjournalpublisherbynumberofjournaltitleswasSpringerNature(includingBioMedCentral),followedbyElsevier(Morrison,2017).Thisdevelopmentisnotananomaly.Identifyingasub-setofjournalsas“core”andthereforemoredesirabletopublishinandmoreessentialtopurchaseincreasestheir

13

marketvalue.“Core”isinquotestoemphasizethatthisisanessentializationoftheconceptformarketpurposes.Theprioritiesoffor-profitpublishersarereturningprofittoshareholdersorprivateowners,notthehealthofthescholarlypublishingecosystem.Thus,itislogicalthatjournalIFexacerbatestheproblemofaffordabilityofscholarlypublishingandsimilarlylogicaltohypothesizethatnewbibliometrics-basedapproacheswillhaveasimilareffect.Qualitativefocusedassessment:how,why,andtheUniversityofOttawaasmodelTherearemodelsforassessingresearchatthelevelofevaluationofindividualresearchers,programs,andinstitutionsthatexemplifyanunderstandingofthebroadervalueofresearchtosocietyandaddressthecomplexityofthediversityofresearchasexpressedintheScienceEuropevisionandtheLeidenManifesto.Atmyownuniversity,theUniversityofOttawa,thecriteriaforevaluatingfacultymembersforpromotionandtenureiscollaborativelydevelopedbyfacultyandadministrationandgovernedbytheAssociationofUniversityProfessorsoftheUniversityofOttawa(APUO)CollectiveAgreement(2016-2018).ThefulltextofallsectionsofthecollectiveagreementdirectlyrelevanttoresearchassessmentcanbefoundintheAppendix.ThelanguageoftheAPUOagreementaddressesthequestionofdiverseandevolvingformsofscholarlyworks.Section23.3.1(h)regardingthetypesofmaterialthatmembersmaysubmitforassessment,states“itisunderstoodthatsincemethodsofdisseminationmayvaryamongdisciplinesandindividuals,disseminationshallnotbelimitedtopublicationinrefereedjournalsoranyparticularformormethods”;oneexampleofotherformsislistedinsection(a),“inthecaseofliteraryorartisticcreation,originalworksandformsofexpression”.Canada’sSocialSciencesandHumanitiesResearchCouncilrecognizes“research-creation”asavalidformofdissemination.Typically,anewfacultymemberattheUniversityofOttawaishiredattherankofAssistantProfessorand,aftersixyears,appliesfortheseniorrankofAssociateProfessor,apromotionthatautomaticallyinvokestenure.ThecriteriaforresearchassessmentatthisstagearecoveredundertheAPUOCollectiveAgreementSection25.3.2.2(c).Toachievetenure,anewfacultymembermustdemonstrateproductionofgoodqualityscientific,literary,artistic,orprofessionalworks,thatgobeyondworkdoneinthecompletionofthedoctorateandthatshowcontinuousprogress.Evaluationisconductedbythreeoutsideevaluatorsandreviewedbycommitteesatthefacultyanduniversity-widelevel.AsimilarprocessisfollowedwhenafacultymemberappliesforpromotiontotherankoffullortitularProfessor.Thisisaholisticcareer-levelpeerreviewprocess.Ironically,andsomewhatmysteriously,inspiteofthisqualitativeandinclusiveapproachtoassessmentofresearchandresearchersratherthansimplisticmetrics,theUniversityofOttawadoesverywellinmetrics-basedrankings.AccordingtotheUniversityofOttawa&GovernmentofOntarioMinistryofAdvancedEducationandSkillsDevelopment2017-2020StrategicMandateAgreement(p.17),“IndependentnationalandInternationalrankings(suchasResearchInfosource,QSWorldUniversityandtheTimesHigherEducation)consistentlyplaceuOttawaamongthetopthreeOntariouniversities,amongCanada’stop10researchuniversitiesandamongthetoptwopercentoftheworld’suniversities”.Whyisthis?Theanswertothisquestionisnoteasytoascertainbyreviewingrankingagencies’descriptionsofmethodology.TimesHigherEducationclaimsforthe2019

14

rankingstohavethefirstauditeduniversityrankingsandappearstobethemosttransparent.TheoverallrankingfortheUniversityofOttawafrom2012to2019variedfromahighof171(2013)toalowof251-300in2017,risingto201–250in2018and176in2019.TheareaswheretheUniversityofOttawaappearstoscorerelativelywellarecitations,industryincome,andinternationaloutlook.Citationscountfor30%oftheoverallweightandsoarelikelythemajorfactor.ThereisnoobviousreasonfromthedetailedmethoddescriptionofTHEastowhytheaveragecitationstoresearchpublishedbyUniversityofOttawafacultywouldhavechangedduringthistimeframe.Overthistimeframe,thefull-facultyfacultycomplementhasdecreasedslightly;overtime,onewouldexpectthistodecreasethenumberofcitationswithlessresearchbeingproduced.Aslightdecreaseinthefacultycomplementwouldaccountforaslightlyhigheraveragecitationrateifthefacultycomplementwerefactoredin,howevertheTHEmethodologydoesnotstatethatthisisthecase.Industryincomemayreflectlocaleconomicconditions.TheUniversityofOttawa’scentrallocationinthenation’scapital(closetodowntown,CityHall,Parliament,surroundedbyembassies,closetonationalcorporateheadoffices)maybeafactorintheuniversity’sstrengthsinindustryincomeandinternationaloutlook.However,thewidevariationinoverallstandingoverrecentyears,givenastableuniversity,suggestthatchangesinhowrankingsarecalculatedarealargefactorincurrentstandings.AttheUniversityofOttawa,regularcomprehensiveassessmentprocessesthatincludepeerassessmentoffacultyandstudentresearcharealreadyinplaceattheprogramlevel.Atminimum,eachprogramundergoesaprovinciallymandatedcyclicalreviewevery7years.Thisisfarmorein-depthassessmentthancountingpublicationsorcitations.Forexample,externalreviewersconductin-personinterviewsseparatelywithfaculty,students,andadministratorsandhavetheopportunitytoaskquestionsnotjustaboutresearchoutputs,butalsoaboutinstitutionalsupportforresearchintermsoftime,facilities,assistancewithgrantapplicationsandsoforth.Inaddition,someprograms,particularlyprofessionalprograms,undergoprofessionalaccreditationprocesses,thatalsoreviewresearchundertakenatthedepartmentallevel.Forexample,theSchoolofInformationStudiesundergoesarigorousaccreditationprocesscoordinatedbytheAmericanLibraryAssociation.Forauniversitywiththesein-depth,holisticresearchassessmentpracticesalreadyinplace,aturntogreaterrelianceonsimplisticmetricsbasedonalimitedandbackwards-lookingunderstandingofformatsandwhatconstitutesgoodqualityscholarlyworkwouldbeastepbackwards.TheprimarymissionofElsevierisreturningprofittoshareholdersofitsparentcompanyRELX;theprimarymissionofClarivateisreturningprofittoitsprivateowners.ThemissionoftheUniversityofOttawa(2017-2020),withrespecttoitsroleasaresearch-intensiveuniversity:“weprovideourstudentswithanoutstandingeducationandenrichtheintellectual,economicandculturallifeofCanada,helpingourcountryplayanimportantandvaluedroleamongthenationsoftheworld”.Themissionofeachuniversitywilldifferslightly,butwilltendtorevolvearoundthecentralfunctionsofteaching,researchasanactivitydesignedtofurtherourcollectiveknowledge,andservicetotheacademyandtosocietyasawhole.Iarguethatweshouldtrustscholarsandtheacademytodesignandimplementassessmentmechanismsthatreflectandprioritizeourgoals(missions,vision),notthoseofoutsidepartieswhoseprimaryinterestsareinherentlydifferentfromourown.

15

Conclusion:theirrationalrationalityofmetrics-basedassessmentofresearchItislogicalforpeopletowanttomeasureprogresstowardsthegoalsthatwedesire.Manymeasuresarevalidandlogical.However,whenwefocusonthemeasuresperseratherthanthegoals,wecanendupwithresultsthatdonotachieveourgoals.ThisiswhatIcallasuperficiallyrational(mathematical,calculating)approachthatisactuallyirrationalintermsofwhatweareactuallyattemptingtoachieve,orirrationalrationality.Currentandemergingformsofmetrics-basedassessmentofresearchandresearchersdisplaymajorproblemswithirrationalrationalitythatcreateincentivesthatarenotcompatiblewithagoalofproducinganddisseminatingqualityresearch.Theseproblemsmeriturgentattentionbeforeourcurrentfixationwithmetricsfurtherentrenchesexistingproblemsandneworaltmetricsintroducesnewones.Therearevalid,logicalreasonsforuseofsomemetricsinassessingresearch,researchers,researchinstitutions,andpublishers.Auniversityshouldbeabletopointtoasubstantialcorpsoffacultywitharesearchmandateandabodyofresearchworksproducedbyitsfacultytocallitselfaresearchuniversity.Anindividualresearchershouldbeabletopointtoacollectionofpublishedworksand/orsubstantivework-in-progressinordertobeconsideredaproductiveresearcher.Individualresearchersandresearchteamsmayfindithelpfultodevelopspecificmeasurablegoalsthatmakesensefortheirownprojects.Journalsandotherpublisherscanusemetricstoassessmarketingefforts.Bibliometricsisausefulresearchmethodforgeneratingnewknowledge.However,justbecausesomemetricsarehelpful,itdoesnotfollowthatubiquitousmetricsarehelpful.Abitofsaltaddsflavortofood;excesssodiumcauseshighbloodpressure,increasingtheriskofheartattackorstroke.Evaluationbasedonmetricslooksscientific,doesn’tit?Numbersareobjective.Metrics-basedevaluationisrationalandcalculating;scientistsoftenuselotsofdata.However,theresemblanceissuperficial.Logicisapowerfultool;butthevalidityoflogicalargumentsdependsonthevalidityoftheunderlyingassumptions.Inordertoassesswhetherwearemakingprogressinscience,weneedtounderstandhowscienceworks.Asdiscussedabove,ourcurrentapproachestosciencearecompatiblewiththeproductionofdangerouslyerroneous“facts”suchastheequationofvaccineswithautismandfalsebeliefinthesafetyofdrugs;irreproducibleresearch;and,atendencytociteliteraturereviewsthatmakesmewonderhowoftentheoriginalstudiesareactuallyread.Ifthisisthesituationinscience,whataboutotherbranchesofknowledge?AsCamic,Gross&Lamont(2011)discuss,whiletherehasbeensomereflectiononpracticeinthesciencessincepublicationofKuhn’s(1962)TheStructureofScientificRevolutionsinthe1960’sandsubsequentdevelopmentofthefieldofscienceandtechnologystudies(STS),parallelstudyofprocessesintheareaofthesocialsciencesisjustbeginning.Ifwedonotevenknowwhatscholarsinthesocialsciencesdo,howcanweclaimtoknowhowtomeasurewhethertheyaredoingitwell?Thestudyofphilosophyandpracticeofscience,whilemoreadvancedthanthestudyofsocialsciences,raisesmorequestionsthananswersaboutmetrics-basedapproaches.Forexample,itislogicaltoassumethattheparadigmshiftsdescribedbyKuhn(1962)willleadtosituationswherewhetherworksarecited,andbywhom,dependsonthephaseofdevelopmentofnewideas.Onemighthypothesizethatworksthatfitacurrentparadigmwillbecitedmorethanpioneeringworks,inwhichcaseusingcitationstoassessthevalueofresearchwilltendtoincentivizeconservatismoverinnovation.

16

IarguethatKuhn’sworkitselfillustratestheproblem.ThomasKuhnwasayoung,white,highlyeducatedmalebasedintheUnitedStateswhowroteaboutscientificrevolutionsinthe1960’s.Thisworkwaswidelyread,studied,andcited,withinashorttimeafterpublication.Anearlierwork,Fleck’s1930’sground-breakingGenesisofascientificfact(Fleck,1979),didnotenjoythisimmediateacclaim.Intheforewordtothe1979edition,Kuhndescribesfindingthiswork,writteninGerman,byhappenstancewhilebrowsinginthestacksofalibrary.FleckwasaJewishintellectuallackingformalcredentialswhoseworkwaspublishedinGermanyinthe1930’s.UnlikethepopularreceptionofKuhn’swork,only600copieswereprintedofFleck’swork,only200weresold,andonly6weredeliveredtotheUnitedStates.Kuhn’sphilosophywasinspiredbythiswork,butthisdoesnotrendertheworkobsoleteasKuhn’sideascomplementratherthansupersedethoseofFleck.Forexample,whereKuhnemphasizedsuddenrupturesinscientificthinking,Fleckemphasizedcontinuityofbasicpremisesinapparentlyrevolutionaryadvancesinknowledge.Whatdoesthishavetodowithuniversityrankings?Ihavedescribedadeeplyflawedsystem,withanillusoryappearanceofscientificbasis,thatincentivizesquantityofproductionofresearchworksoverquality,noveltyoverrigorouscritiqueandreplication.Themetricsbehindthissystemfeedintouniversityrankings,andtherankingsreinforcethistrendtowardirrationalrationality.Thecurrenttrendtowardsneworaltmetricswillcreateevenmoreirrationalrationality.Itislogicaltoexpectthatthesenewmetrics,particularlymetricsthatdonotdependonacademiccitations,willamplifyexistingproblemswithmetrics-basedevaluationand/orcreatenewones.Ipredictthatsuchmetricswillreflectpre-existingsocialbiases.Theextenttowhichindividualworksarecited,downloaded,and/orsharedviasocialmediaarelikelytocorrelatewithgenderandethnicbiasesaswellasthepopularityoftopicsstudied.Inaddition,metricsthatdonotdependonacademiccitations(downloads,tweets,etc.)arefarmorevulnerabletodeliberatemanipulation.Thefossilfuelindustrycanaffordtohirepeopletodownloadandtweetevidenceofclimatechangedenial,forexample.Anotherfactorthatshouldbeconsideredbeforeusingsuchdataasasurrogateforqualityofresearchistheimpactthatusageofsuchmetricscouldhaveontheresearchitself.Forexample,ifcancerresearchersfindithelpfultousesocialmedia,theycanandshoulddoso.Butifmetricsbasedonnon-academicuseweretoformthebasisofassessmentandresearchinfuture,thiscouldresultinaredirectionofeffortsfromcancerresearchtosocialmediasharing.Asfortheimpactofpopularityoftopicsinnewmetrics,onacynicalnoteItakecomfortinthepossibilitythatsomedayImayhavereasontomoveforwardwithastudyalongthelinesof“correlatesofperceivedattractivenessofjuvenilefelinesonYouTube”[academeseforhowcomethosekittycatsonYouTubearesodarncute]toprovemyworthinessasaresearcher.Onaseriousnote,itismyexperienceasalongtimepractitionerofopenresearchthatthepopularityofmyworksdoesnotcorrelatewiththeimportanceofitscontribution.Myground-breakingbookchapter“Theimplicationsofusagestatisticsasaneconomicfactorinscholarlycommunications”beginssomeofthediscussionthatcontinueswiththischapterandintroducesimportantbutcounter-intuitiveideas(Morrison,2005).ThisworkdoesnotenjoyevenasmallfractionofthesocialmediapopularityofmyDramaticGrowthofOpenAccessblogseries,designedtosupporttheadvocacyeffortsofaglobalmovement.

17

Theevolutionofresearchmetricsdescribedinthischaptercapturestheirrationalrationalityofmetrics-basedresearchassessment.Inthe1950’sand1960’swedevelopedtoolstohelpresearchersandlibrariescopewiththeever-expandingvolumeofscholarlyliteraturebyconnectingcitingandcitedworksandidentifyinghighlycitedjournalstohelplibrarieswithdecisionsaboutpurchasesandcancellations,andresearcherswithdecisionsaboutreadingandpublishing.Theresultingmetric,IF,becameayardstickforevaluatingtheworthinessofresearchandresearchersevenwhenitwasacknowledgebyexpertsandtheproducingcompanythatthismetricwasnotatallsuitableforthispurpose.AttemptstoaddressthetechnicalflawsofIF(connectedtojournalsratherthanarticles)areaddinganewlayerofmetricsbasedoncitationstoindividualworksthatrankingcompaniesarealreadyincorporatingintoassessmentofuniversitiesandthatarebeingmarketedasameansofassessingresearchers.Theprofitgoalsofmetricsbasedcompanies(scholarlypublishers,citationmetricsandrankingservices)areovertakingtheresearchmissionsofuniversities.Wedeveloptoolstohelpusachieveourgoals,thenwebecomeslavestothetools.Thatisirrationalrationality.Inthefuture,ifwecontinueonthecurrenttrajectory,weshouldexpectanadditionallayerofmuchmoreillogicalmetrics-basedcontrolofresearchandresearchersintheformofaltmetricsbasedonusagebeyondtheacademy.Ifexcessrelianceonmetricsistheproblem,whatistheremedy?Let’sdevelopandusemetricswheretheymakesense,basedonthegoalsofindividualresearchprojectsandinstitutions.Butlet’sdosowithagrainofsaltandnotrelyonmetricswheresuchrelianceisnotscientificandmaybecounter-productivebycreatingperverseincentivesforquantityandnoveltyoverqualityandfavourparticularformats,evenastheyarebecomeobsolete.Howcanweimplementthisremedy?Wecanuseapproachesthatappropriatelyweightqualityandthatrecognizethediverseformsofresearch.Insteadoftranslatingaresearchdossierintoun-scientificmetrics,readandreviewtheworks.Thisisn’tnew,andshouldn’tbehard.Thisiswhatwedonowwhenweassessathesisorpeer-reviewtheworksofotherresearchers.AttheUniversityofOttawa,wehaveacollectiveagreementthatacknowledgesthediversityofresearchanditsproducts;detailsareintheAppendixifanyonewouldliketoconsiderthisasonepotentialmodelforchange.ReferencesAPUOCollectiveAgreement(2016-2018).RetrievedJanuary25,2018from

http://www.apuo.ca/collective-agreement/collective-agreement/AssociationofResearchLibraries(ARL).(1989).ReportoftheARLserialspricesproject:A

compilationofreportsexaminingtheserialspricesproblem.Washington,DC:TheAssociationofResearchLibraries.RetrievedJanuary15,2018fromhttp://catalog.hathitrust.org/Record/001527850

Baker,M.(2016).1,500scientistsliftthelidonreproducibility.Nature533,452–454

(26May2016)doi:10.1038/533452aBruckner,Till,&Ellis,Beth.(2017).Clinicaltrialtransparency:akeytobetterandsafer

medicines.RetrievedAugust1,2017fromhttps://media.wix.com/ugd/01f35d_0f2955eb88e34c02b82d886c528efeb4.pdf

Camic,C.;Gross,N.,Lamont,M.(2011).Socialknowledgeinthemaking.Chicago:The

UniversityofChicagoPress.

18

Casadevall,A.&Fang,F.(2010).Reproduciblescience.InfectionandImmunity78:12.

doi:10.1128/IAI.00908-10.RetrievedMay2,2017fromhttp://iai.asm.org/content/78/12/4972.full

ClarivateAnalytics(n.d.).Products:JournalCitationReports.RetrievedJanuary15,

2018fromhttps://clarivate.com/

Dwyer,M.(2017).Measuringexcellence:howwerankCanada’suniversities.Maclean’s.RetrievedJanuary23,2018fromhttp://www.macleans.ca/education/measuring-excellence-how-we-rank-canadas-universities/

ElsevierScopus(n.d.).RetrievedJanuary23,2018fromhttps://www.elsevier.com/solutions/scopus

European Commission (2018). Plan S and cOAlition S – accelerating the transition to full

and immediate Open Access to scientific publications. (Statement). Retrieved Feb. 18, 2019 from https://ec.europa.eu/commission/commissioners/2014-2019/moedas/announcements/plan-s-and-coalition-s-accelerating-transition-full-and-immediate-open-access-scientific_en

Fleck, L. (1979.). Genesis and development of a scientific fact. (T. Trenn & R.K. Merton,

Eds; T. Trenn & F. Bradley, Trans.) Foreword by Kuhn, T.S. Chicago : University of Chicago Press.

Garfield(1955).Citationindexestoscience:anewdimensionindocumentationthrough

associationofideas.Science.1955:122:108-111.RetrievedJanuary15,2018fromhttp://garfield.library.upenn.edu/essays/v6p468y1983.pdf.

Garfield,E.(2006).Thehistoryandmeaningofthejournalimpactfactor.Journalofthe

AmericanMedicalAssociation295:1.

Guédon,J.C.(2001).InOldenburg'slongshadow.Washington,D.C.:AssociationofResearchLibraries.

Haustein,S.(2012).Multidimensionaljournalevaluation:analyzingscientificperiodicals

beyondtheimpactfactor.Berlin:DeGruyter/Saur.Hicks,D.;Wouters,P.;Waltman,L.;DeRijcke,S.&Rafois,I.(2015).Bibliometrics:theLeiden

Manifestoforresearchmetrics.Nature520,429–431(23April2015)doi:10.1038/520429a.

Khodiyar,V.K.,Rowlett,K.A.andLawrence,R.N.(2014).Altmetricsasameansofassessingscholarlyoutput.LearnedPublishing,27:S25–S32.doi:10.1087/20140505

Kuhn,T.(1962).Thestructureofscientificrevolutions.Chicago:UniversityofChicagoPress.LaRivière,V.;Haustein,S.&Mongeon,P.(2015).Theoligopolyofacademicpublishers

19

inthedigitalera.PLOSONEJune15,2015.RetrievedJanuary15,2018fromhttps://doi.org/10.1371/journal.pone.0127502

Mabe,M.(2003).Thegrowthandnumberofjournals.Serials,16(2),191-197.Mabe,M.,&Amin,M.(2001).Growthdynamicsofscholarlyandscientificjournals.

Scientometrics,51(1),147-162.Mahood,S.(2015).Privatizedknowledgeandthepharmaceuticalindustry.In:Elliott,P.&

Hepting,D.FreeKnowledge:confrontingthecommodificationofhumandiscovery,pp.26-38.Regina:UniversityofReginaPress.

Mignolo, Walter D. (2002). The Geopolitics of Knowledge and the Colonial Difference.

South Atlantic Quarterly, 101(1), 57-96. Morrison,H.(2005).Theimplicationsofusagestatisticsasaneconomicfactorinscholarly

communications.In:Usagestatisticsofe-resources,D.Fowler(ed.).HaworthPress.Morrison,H.(2012).Thecommercializationandrationalizationofscholarlypublication.In:

Morrison,H.Freedomforscholarshipintheinternetage.Doctoraldissertation,SimonFraserUniversity.RetrievedJan.15,2018fromhttp://summit.sfu.ca/item/12537

Morrison,H.;Brutus,W.;Dumais-Desrosier,M.;Laprade,K.;Merhi,S.;Ouerghi,A.;Salhab,J.;

Volkanova,V.;Wheatley,S.(2017),Openaccessarticleprocessingcharges2016,doi:10.5683/SP/KC2NBV,ScholarsPortalDataverse.

Morrison,H.(2017).Elsevierasanopenaccesspublisher.TheCharlestonAdvisor18:3,pp.

53–59.RetrievedJanuary15,2018fromhttp://www.ingentaconnect.com/content/charleston/chadv/2017/00000018/00000003/art00014

Morrison,H.;Brutus,W.;Dumais-Desrosiers,M.;Kakou,T.L.;Laprade,K.;Merhi,S.;Ouerghi,

A.;Salhab,J.;Volkanova,V.;Wheatley,S.(2017).OpenAccessArticleProcessingCharges(OAAPC)LongitudinalStudy2016Dataset.Data2:13.

Odlyzko,A.M.(1994).Tragiclossorgoodriddance?theimpendingdemiseofscholarly

journals.JournalofUniversalComputerScience,0(0),1-52. Price,D.J.d.S.(1963).Littlescience,bigscience.NewYork:ColumbiaUniversityPress.

QSWorldUniversityRankings(2018).Methodology.RetrievedJanuary22,2018fromhttps://www.topuniversities.com/qs-world-university-rankings/methodology

RetractionWatch(n.d.).Top10mosthighlycitedretractedpapers.Website.RetrievedJanuary22,2018fromhttp://retractionwatch.com/the-retraction-watch-leaderboard/top-10-most-highly-cited-retracted-papers/

SanFranciscoDeclarationonResearchAssessment(2012).RetrievedJanuary15,2018

fromhttp://www.ascb.org/dora/

20

Schafer,A.(2015).Pseudo-evidence-basedmedicine:whenbiomedicalresearchbecomesanadjunctofpharmaceuticalmarketing.In:Elliott,P.&Hepting,D.FreeKnowledge:confrontingthecommodificationofhumandiscovery,pp.39-55.Regina:UniversityofReginaPress.

Schiltz,M.(2018).WhyPlanS?RetrievedFeb.18,2019fromhttps://www.coalition-s.org/why-plan-s/

ScholarlyPublishingandAcademicResourcesCoalition(SPARC)(n.d.).Bigdealcancellation

tracking.Website.RetrievedJanuary15,2018fromhttps://sparcopen.org/our-work/big-deal-cancellation-tracking/

ScienceEurope(2017a).ScienceEuropepositionstatement:onanewvisionformore

meaningfulresearchassessment.RetrievedJanuary15,2018fromhttp://www.scienceeurope.org/tallinn-call-for-action-takes-up-science-europe-recommendation-on-research-impact/

ScienceEurope(2017g).TaillinnCallforAction.RetrievedJanuary17,2018fromhttp://www.scienceeurope.org/tallinn-call-for-action-takes-up-science-europe-recommendation-on-research-impact/

Smith,R.(2006).Thetroublewithmedicaljournals.London:RoyalSocietyofMedicine

Press.Stephan,P.;Veugelers,R.&Wang,J.(2017).Reviewersareblinkeredbybibliometrics.

Nature544,p.411-412(27April2017)doi:10.1038/544411a.TimesHigherEducation(2018).WorldUniversityRankings2018.Inpartnershipwith

Elsevier.RetrievedJan.22,2018fromhttps://www.timeshighereducation.com/world-university-rankings/2018/world-ranking

U.K.OfficeofFairTrading.(2002).Themarketforscientific,medicalandtechnicaljournals.No.OFT396U.K.OfficeofFairTrade.RetrievedSeptember13,2011fromhttp://www.oft.gov.uk/advice_and_resources/publications/reports/media/

UniversityofOttawa&GovernmentofOntarioMinistryofAdvancedEducationandSkills

Development(2017).StrategicMandateAgreement2017-2020.Ware,M.&Mabe,M.(2015).TheSTMreport:anoverviewofscientificandscholarly

journalpublishing.4thed.TheInternationalAssociationofMedical,TechnicalandScientificPublishers(STM).RetrievedMay2,2017fromhttp://www.stm-assoc.org/2015_02_20_STM_Report_2015.pdf

21

AppendixTheAssociationofUniversityProfessorsoftheUniversityofOttawa(APUO)CollectiveAgreement(2016-2018)section23.3(assessmentofscholarlyactivities)states:

“23.3.1(a)Themembermaysubmitforassessmentarticles,booksorcontributionstobooks,thetextofpresentationsatconferences,reports,portionsofworksinprogress,and,inthecaseofliteraryorartisticcreation,originalworksandformsofexpression.(h)Itisunderstoodthatsincemethodsofdisseminationmayvaryamongdisciplinesandindividuals,disseminationshallnotbelimitedtopublicationinrefereedjournalsoranyparticularformormethods”.

Typically,anewfacultymemberishiredattherankofAssistantProfessorand,aftersixyears,appliesfortheseniorrankofAssociateProfessor,apromotionthatautomaticallyinvokestenure.ThecriteriaforresearchassessmentatthisstageiscoveredundertheAPUOCollectiveAgreementSection25.3.2.2(c)andreadsasfollows:

“(c)TheMemberhasproducedscientific,literary,artistic,orprofessionalworks--oracombinationthereof--whichare,inaccordancewiththecriteriasetforthin23.3.3.2,deemedofgoodquality.ThisassessmentshallbemadefollowinganoverallevaluationoftheMember'sscholarlyworks,carriedoutinaccordancewiththeprovisionsofsection23.3,duringwhichtheopinionofthree(3)outsideevaluatorswillhavebeenobtained,inaccordancewith23.3.2”.

23.3.3Levelofperformanceofscholarlyactivitiesstates:

*23.3.3.1WheneverthisagreementreferstosatisfactoryperformanceofscholarlyactivitiesbyaFacultyMember,itreferstoasituationwheretheMemberisregularlyengagedinscholarlyactivitiestheresultsofwhichindicatethatherperformance,incomparisontoarelevantgroupofpeersofcomparablerankandexperience,issatisfactory.*23.3.3.2TheMember'sscholarlyworksshallbeconsideredgoodiftheyrepresentacontributioninadditiontothatcontainedintheMember'sdoctoralthesisortotheworkthathasbeentakentobetheequivalentofadoctorate,andif,subsequenttothatwork:(a)inthecaseofresearch,theydemonstratecontinuousprogressinthedevelopmentoftheMember'sresearchactivitiesandcontributetotheadvancementofknowledgeintheMember'sfieldofspecialization;(b)inthecaseofliteraryorartisticworks,theyattesttocontinuouscreativeactivity,well-reputedintheliteraryorartisticcommunityoutsidetheUniversityofOttawa;(c)inthecaseofprofessionalworks,theyattesttothepracticeofaprofessionaboveandbeyondthatwhichisgenerallyexpectedofanon-teaching,practicingprofessional,ortheycanbeconsideredasavaluablecontributiontotheadvancementoftheprofessionitself.