Measuring quality of research: what do they mean and why they mean so?

  • Published on

  • View

  • Download

Embed Size (px)


<ul><li><p>Computational Biology and Chemistry 28 (2004) 163164</p><p>Editorial</p><p>Measuring quality of research: what do they mean andwhy they mean so?</p><p>Quantitative judgment of outcomes of creative work inscience sounds like an oxymoron from the outset. Yet scoressuch as citation index for scientific papers or the impact fac-tor of journals are routinely applied by the bureaucracy ofscientific establishments to rate quality of creative humans.This form of numerology naturally induces resentment in thescientific community with the exception of a vast minorityof savants who were rewarded for publishing large numbersof frequently cited papers in high impact journals. Per-haps the most misleading aspect of citation-based indices ishigh number of citations of papers that are methodologicallyflawed and therefore criticized by many readers. If for in-stance savant A published a paper that was cited by 25 otherpapers in a positive way there is no reason to dislike numer-ical score 25 applied to scholar A. However, if anotherscholar B published a paper that was criticized for funda-mental methodological errors in 20 other papers while men-tioned positively in five more papers the number of citationsfor B is also 25. The obvious problem here is that the authorof a solid methodological paper (scientist A) has acquired thesame numerical score (of 25) as researcher B who publisheda methodologically flawed work with no explanatory or pre-dictive value. Albeit this situation appears unacceptable it isnot so for numerous tenure committees who take numericalscores as the basis for promotion. One can easily imaginethat if scientists A and B compete for a position in the sameinstitution it is enough for scholar B to acquire one moreinstance of critique (21st) to win the competition 2625.</p><p>Interestingly, the foregoing misgivings could be very eas-ily reduced with relatively simple modifications in numericalscoring. One obvious such modification would be markingeach citation with one of three labels: positive, negative,and neutral (or neither). One could then, as it is often doneoutside science, call positive a +1, negative a 1 andneutral a 0. In the next step instead of adding citations onecould add their labels. This procedure applied to our exam-ple would result in savant A having a score 25 (as previ-ously) but scholar B would end up with the score of15 (thedifference of 40 points). According to this measure scientistA would properly appear as a better one than researcher B.Needless to say impact factor of journals could be seriouslycorrected as well by using modified scoring of citations.</p><p>I hasten to add that even more complex numerical scoringsystems will not give justice to several aspects of a given sci-entific paper that inspire other work. For instance, the styleof writing, selection of metaphors and level of enthusiasm ofthe author do impact other work. So do personal discussionsand meeting presentations. That brings yet another dimen-sion to the rating problem: What is the actual motivationof the impacted? Some of them simply share enthusiasm ofimpacting author some others believe that imitation of orig-inal (i.e. impacting) work should immediately be providedto the community. Yet another group of impacted scientistsmay see the analogy of their field with the field of impactingauthor.</p><p>As far as Computational Biology and Chemistry is con-cerned the editorial board is focused on four primary indica-tives of papers quality:</p><p>1. Novelty in terms of entirely unexpected, new, researchquestion(s) or unexpected outcomes of data analyses(both routine and novel).</p><p>2. Originality in terms of style and erudition associated withformulating research question(s) and interpretation ofresults.</p><p>3. Conformity to state-of-the-art in computational life sci-ences and its composite disciplines. In particular, theomissions of pivotal literature in a manuscript will nolonger be acceptable. Nor misinterpretation via mislead-ing citation(s) will be allowed.</p><p>4. Potential for reproducibility: The paper needs to be writ-ten in a way that will allow one to reproduce its resultsgiven sufficient time, skills, and level of interest in theproblem.</p><p>We still do not measure the intellectual potential of theauthors approach to writing itself but we will consider thisfactor in the future. For the moment we do accept only thepapers which the editorial panel considers methodologi-cally solid at least in terms of points 3 and 4 above. Howdo our criteria relate to the numerical ones (such as numberof citations)? Indirectly, they relate through point 3 aboveas completeness and accuracy of citations are required foracceptance of manuscripts to CBAC. However, the three re-maining criteria of quality do not seem to have much to do</p><p>1476-9271/$ see front matter 2004 Published by Elsevier Ltd.doi:10.1016/j.compbiolchem.2004.06.001</p></li><li><p>164 Editorial</p><p>with citation index unless we misjudge by mistake the nov-elty and originality of a manuscript. What seems to counthere much more than the number of citations is the under-standing of authors meanings by the referees and the read-ers. If what the authors mean and the reasons why they choseto mean so appear to be inspiring for the skilled readers, thepaper can be rated as high quality contribution. Of course thecategory of skilled readers does not seem to be definablequantitatively. Thereby it would be unwise to expect thatquantitative criteria for quality of papers will ever replace</p><p>human evaluation. The only feasible way for such evalua-tion to be objective is to assume that the editorial panel con-sists of well educated experts with interests similar to thoseof the authors and with passion to exercise unusually highmoral standards of fairness and truthfulness.</p><p>Andrzej K. KonopkaBioLingua Research Inc., CASA Center, 10331 Battleridge</p><p>Place, Gaithersburg, MD 20886, USAE-mail address:</p></li></ul>