Reviewing Papers: What Reviewing Papers: What Reviewers Look ForReviewers Look For

Session 19Session 19

C507C507

Scientific WritingScientific Writing

Are Referees really Necessary?Are Referees really Necessary?

Some look at reviewers (referees) solely Some look at reviewers (referees) solely as biased adversaries whose objectives as biased adversaries whose objectives are solely to reject, delay or scoop all are solely to reject, delay or scoop all papers submitted to thempapers submitted to them

Some feel that science would do quite well Some feel that science would do quite well without reviewers at all, that we could without reviewers at all, that we could publish all readable papers without delay publish all readable papers without delay and let the community at large sort the bad and let the community at large sort the bad from goodfrom good

Are Referees really Necessary?Are Referees really Necessary?

For those that argue against the review For those that argue against the review process and for unrestricted access, they process and for unrestricted access, they fail to distinguish between the formal and fail to distinguish between the formal and informal systems of communication in informal systems of communication in sciencescience

Price SaysPrice Says

““The scientist is more frequently than not The scientist is more frequently than not passionate, biased, illogical, resistant to passionate, biased, illogical, resistant to proof and to change, and beset by other proof and to change, and beset by other similar human failings. It is, today, better to similar human failings. It is, today, better to let this show and be understood than to let this show and be understood than to pretend that it is not there.”pretend that it is not there.”

An Overlooked FunctionAn Overlooked Function

One function of the peer review system One function of the peer review system that is often overlooked is its indirect that is often overlooked is its indirect influence on the initial preparation of a influence on the initial preparation of a paperpaper

Established scientists write their papers Established scientists write their papers with a critical sense that anticipates peer with a critical sense that anticipates peer reviewer questionsreviewer questions

Does the System Work?Does the System Work?

The answer to this question seems to be a The answer to this question seems to be a resounding “Yes.”resounding “Yes.”

All editors, and most authors will affirm All editors, and most authors will affirm that there is hardly a paper published that that there is hardly a paper published that has not been improved, often has not been improved, often substantially, by the reviewerssubstantially, by the reviewers

These same editors will also attest that These same editors will also attest that examples of intentional delay, biased examples of intentional delay, biased reporting or unethical behavior is rarereporting or unethical behavior is rare

The Different Processes of Peer The Different Processes of Peer ReviewReview

Evaluations of papers for publicationEvaluations of papers for publication Assessment of grant proposalsAssessment of grant proposals Recommendations for positions or tenureRecommendations for positions or tenure

Two Kinds of ConsensusTwo Kinds of Consensus

There are times when referees will There are times when referees will disagree about the suitability of a paper for disagree about the suitability of a paper for publication, and so will the journal editorpublication, and so will the journal editor

Not infrequently, a paper rejected by one Not infrequently, a paper rejected by one journal is accepted by anotherjournal is accepted by another

These disagreements do not indicate a These disagreements do not indicate a breakdown of the referee system; rather, breakdown of the referee system; rather, they arise from differing views of two kinds they arise from differing views of two kinds of consensusof consensus

Consensus #1Consensus #1

First, there is the consensus that develops First, there is the consensus that develops within an editorial board on what within an editorial board on what constitutes an acceptable paper constitutes an acceptable paper for that for that journaljournal

Consensus #2Consensus #2

The second consensus that enters into the The second consensus that enters into the evaluation of a scientific paper is the one evaluation of a scientific paper is the one that develops within a field of researchthat develops within a field of research

In some areas of science the fundamental In some areas of science the fundamental algorithms and paradigms are well algorithms and paradigms are well established, tested, verified and acceptedestablished, tested, verified and accepted

Possible SolutionsPossible Solutions

You could publish the paper and include You could publish the paper and include the reviewer’s comments as a comment or the reviewer’s comments as a comment or addendum to the paperaddendum to the paper

Or you could consider open review- here, Or you could consider open review- here, you seek comments from several referees you seek comments from several referees to be published concurrently with the to be published concurrently with the paperpaper

Does Reviewer Anonymity Matter?Does Reviewer Anonymity Matter?

This remains controversialThis remains controversial But, by and large, reviewers really do try But, by and large, reviewers really do try

their best to offer useful comments to their best to offer useful comments to authors. authors.

Reviewers need to report with candor and Reviewers need to report with candor and honesty; keep them anonymous is often honesty; keep them anonymous is often believed to help do just thatbelieved to help do just that

Blind ReviewBlind Review

We also remove the author’s name from a We also remove the author’s name from a paper sent out for reviewpaper sent out for review

This is supposed to reduce biasThis is supposed to reduce bias But this really does not work too well, just But this really does not work too well, just

in terms of keeping identity confidential; in terms of keeping identity confidential; we have too small a population of we have too small a population of researchers in our professionresearchers in our profession

ReviewersReviewers

In general, journals send papers to at least In general, journals send papers to at least 2 reviewers, but may at times use more2 reviewers, but may at times use more

Reviewers are selected because:Reviewers are selected because: They have expertise of use to the journalThey have expertise of use to the journal They support the goals and mission of the They support the goals and mission of the

journaljournal

General Questions Reviewers General Questions Reviewers Need to AnswerNeed to Answer

Does this paper report a specific, Does this paper report a specific, identifiable advance in knowledge?identifiable advance in knowledge?

Has the work reported in this paper been Has the work reported in this paper been published before?published before?

Are the conclusions justified, soundly Are the conclusions justified, soundly based and logically consistent?based and logically consistent?

General Questions Reviewers General Questions Reviewers Need to AnswerNeed to Answer

Are the procedures and methods Are the procedures and methods sufficiently clear that the work could be sufficiently clear that the work could be repeated by anyone knowledgeable in the repeated by anyone knowledgeable in the field?field?

Are the references to previous work Are the references to previous work pertinent and complete?pertinent and complete?

General Questions Reviewers General Questions Reviewers Need to AnswerNeed to Answer

Is the paper as concise as it could be, Is the paper as concise as it could be, consistent with clarity?consistent with clarity?

Referees are not required to comment on Referees are not required to comment on writing style. However, they are invited to writing style. However, they are invited to suggest changes that would remove suggest changes that would remove ambiguity or clarify meaningambiguity or clarify meaning

General Questions Reviewers General Questions Reviewers Need to AnswerNeed to Answer

Are the figures and tables relevant and Are the figures and tables relevant and properly prepared?properly prepared?

Are the title and abstract truly descriptive Are the title and abstract truly descriptive of the contents?of the contents?

Ethics for ReviewersEthics for Reviewers

We remind reviewers that the paper under We remind reviewers that the paper under consideration is a confidential document that consideration is a confidential document that should not be discusses or shown to others should not be discusses or shown to others without the express permission of the editorwithout the express permission of the editor

As the reviewer’s detailed report may be relayed As the reviewer’s detailed report may be relayed to the author as a guide for revising their paper, to the author as a guide for revising their paper, we request that you avoid harsh or abrasive we request that you avoid harsh or abrasive statements (comments specifically to the editor statements (comments specifically to the editor should be sent as a separate written letter)should be sent as a separate written letter)

Ethics for ReviewersEthics for Reviewers

Your anonymity as a reviewer will be Your anonymity as a reviewer will be preserved and you are asked not to preserved and you are asked not to identify yourself to the authors without the identify yourself to the authors without the permission of the editorpermission of the editor

The paper must be returned within a The paper must be returned within a specified period of timespecified period of time

The JMPTThe JMPT

Let us say that a new paper is submitted Let us say that a new paper is submitted to the JMPTto the JMPT

We acknowledge receipt of the paper on We acknowledge receipt of the paper on the day it is received, and a letter that the day it is received, and a letter that “accepts the paper for consideration to “accepts the paper for consideration to publish” is sent to the principal authorpublish” is sent to the principal author

The JMPTThe JMPT

Then, we ready two copies to go out for Then, we ready two copies to go out for reviewreview

The paper is blindedThe paper is blinded Two reviewers, with expertise in the Two reviewers, with expertise in the

subject are and for whom we believe to subject are and for whom we believe to have no bias, are selectedhave no bias, are selected

Appropriate forms are sent with the paperAppropriate forms are sent with the paper

The JMPTThe JMPT

Reviewers then provide their critiqueReviewers then provide their critique They are asked to return their review They are asked to return their review

within 2 weeks. 10% might actually do sowithin 2 weeks. 10% might actually do so The editor uses those reviews as a guide The editor uses those reviews as a guide

to his or her decisionto his or her decision Most frequently, the paper is returned for Most frequently, the paper is returned for

revisionrevision

The JMPTThe JMPT

On the review form, the reviewer can On the review form, the reviewer can check off a set of good points, bad points, check off a set of good points, bad points, and then offer a comment on publishability and then offer a comment on publishability and/or recommendations for improvementand/or recommendations for improvement

The JMPTThe JMPT

Good PointsGood Points OriginalOriginal TimelyTimely Interesting to readInteresting to read PracticalPractical Information important to documentInformation important to document Well suited to journal’s objectivesWell suited to journal’s objectives Good researchGood research Good review of literatureGood review of literature Well written/editedWell written/edited Other good pointsOther good points

Bad PointsBad Points Too preliminaryToo preliminary Duplicates other published research- no new insightDuplicates other published research- no new insight Not well suited to journal’s objectivesNot well suited to journal’s objectives Faulty research designFaulty research design Poorly supported by referencesPoorly supported by references Highly speculativeHighly speculative Plagiarized materialPlagiarized material Not well written/editedNot well written/edited Other bad pointsOther bad points