1

How to review a paper

by

Fabio Crestani

2

Disclaimer

There is no fixed mechanism for refereeing There are simple rules that help transforming

a review in a constructive document In time you will develop your own style of

refereeing This talk mostly reflects my style

3

Purpose

A review serves several roles The precise combination varies with the type

of review– Technical/experimental approach and analysis– Computation– Ignorance of related research– Presentation style– Patents or legal issues

In our case: quality control!

4

Method

Who is interested in quality?– Program chair– Journal/book editor– You!

How do you assure quality?– Papers are reviewed by expert in the field– Peer review– You!

5

The processJournal

Area “generalist”(choose appropriatereviewers)

Conference

Area specialist(peer)

Submitted to

Referees

Editor in chief Program chair

Editors / Editorial Board members

Recommendations

Referees

Program committee(meta reviewers)

6

The role of the referee

Advisory– Authors bears responsibility for correctness of

result presentation– Editor bears responsibility for acceptance/rejection

7

Why doing it?

Several reasons– Enhance reputation (with editor/prog. committee)– Expedites processing of your own papers– Get on editorial board or program committee– Good practice

• Increase your own critical appraisal ability• Your papers become better

– Sometimes it gets preferential treatment for your papers

… but refereeing means more work!

8

Consideration

Most reviews have strict deadlines By agreeing to review you take the

responsibility of doing a thorough job If you cannot commit to this, notify the editor

asap Editors understand you may not have the

time, but are unforgiving if you commit and do a poor job

Good editors keep a list …

9

Types of reviews

Anonymous– Standard for journals and conference– Can be “blind” or “double blind”

Friendly– When you ask a colleague to give comments on a

draft– Has limitations …

Internal– Used mostly in commercial labs for patents and

legal issues

10

How to do a review

Plan to read the paper 3 times1. To get a feel for it

2. Read the paper in depth

3. Mark in up

Fill out the review right after the 3rd reading, while things are still fresh in memory

11

Review structure

a. The actual refereeing form

b. General comments on the paper

c. Specific comments on the paper

d. Confidential note to editor

General idea: be professional and non-hostile: write the review in a style that you would like to receive for your paper

12

The refereeing form

Forms might look quite different but basically ask the same things (see examples)

Poorly designed ones just have yes/no answers, good ones prompt the referee to elaborate

Make sure you read and understand it well

13

General comments

Usually starts with 1-3 sentences summarising the paper to show that you understood it

Discuss author’s assumptions, motivations, technical approach, analysis, results, conclusions, references.

Be constructive, suggest improvements

14

Specific comments

Comments on style, figure, grammar, spelling mistakes, etc.

You can mark up directly on the paper or type in list (or bullet points) form, with reference to the page, section, etc.

It is up to you to decide the level of detail of your specific comments

You are not asked to rewrite the paper!

15

Confidential note

Comments to the editor that you do not want the author to see

Not necessary and do not feel obliged Remember the review should mostly help the

author, so do not “hide” comments

16

Outcome

Usually:a. Accept the paper as it is

b. Paper requires minor changes

c. Paper requires major changes (with or without a new refereeing process)

d. Reject publication of the paper

You can only suggest, the choice is not yours– Decision is based on at least 3 reviews

17

What to consider (1)

Correctness– Of argument/method/algorithm/proof

Significance– Rule out the obvious/trivial solutions– Valid problem– Significance to area/journal

Innovation– Original, novel– Not trivial extension or combination of old work

18

What to consider (2)

Interesting– Well motivated– Relevant (when and where)

Timeliness– Of current interest to community– Take into account: publication delay, pre-exposure

(WWW)

Succint– Message should be: clear, compelling, to the point

19

What to consider (3)

Accessible– Is it appropriate to the audience– Readable, good grammar, good structure– People do not have the time to read badly written

papers

20

Ethics of refereeing (1)

Objectivity– Judge paper on its own merits– Remove prejudice– If you are not able to review it, return it

Fairness– Author may have different point of view /

methodology / arguments– Judge from their school of thought not yours

Speed– Be fast, but do not rush. Author deserves a fair

hearing

21

Ethics of refereeing (2)

Professional treatment– Act in the best interest of the author and

conference/journal– Specific rather than vague criticism

Confidentiality– Cannot circulate paper– Cannot use without permission

Conflict of interest– Discuss with editor

22

Ethics of refereeing (3)

Honesty– About your expertise and confidence in appraisal

Courtesy– Constructive criticism– Non-inflammatory language– Suggest improvements

23

Need practice?

Euan Minto Prize– Make sure you read the notes that accompany the

review form

Give your draft paper to a colleague for comments …

… and be ready to provide some on his/her paper!