CMPUT 603 - Teaching and Research Methods

1

CMPUT603 - Fall 2005

Topic2: Refereeing(After Alan J. Smith, “The Task of the Referee”, IEEE Computer, April, 1989, pp. 65-71.)

José Nelson Amaral et al.http://www.cs.ualberta.ca/~c603

CMPUT 603 - Teaching and Research Methods

2

Role of the Referee

What is the role of the referee? For an acceptance recommendation:

Is the referee responsible for the correctness of the paper?

For a rejection:What should be the goal of the report?

Does the quality of the report affect the referee reputation? How?

CMPUT 603 - Teaching and Research Methods

3

Referee Report Structure

Summarize your recommendation Summarize the point of the paper Evaluate the significance of the

research Evaluate the quality of the research

Methodology, techniques, accuracy, presentation

Overall recommendation On a rejection, clearly state the reasons Make the strength of your opinion clear On acceptance, list required and suggested

changes.

CMPUT 603 - Teaching and Research Methods

4

Evaluation of a Research Paper

What is the purpose of the paper?Is the paper appropriate?Is the goal significant?Is the method of approach valid?Is the execution of the research correct?Are the conclusions supported by the data?

Is the presentation satisfactory?What did you learn?

CMPUT 603 - Teaching and Research Methods

5

Evaluation of a Research Paper (categories)

Major resultsGood, solid, interesting workMinor, but positive, contribution to knowledge

Elegant and technically correct but useless

Neither elegant nor useful, but not wrongWrong and misleadingSo badly written that technical evaluation is impossible

CMPUT 603 - Teaching and Research Methods

6

Comparative Evaluation

What are the standards of this journal or conference? If you recommend a revision (minor or major), will you see the paper again?

If you do, what should you check for?

CMPUT 603 - Teaching and Research Methods

7

Evaluating a Tutorial

Are the title and abstract adequate?Is the scope too wide, too narrow, too bizarre?

Does the paper have a consistent theme?

Is the material correct?Is the coverage too simple or too sophisticated?

Is the paper well-written and clear?

CMPUT 603 - Teaching and Research Methods

8

Evaluating a Survey

Is the author an expert in the field?Is the material integrated in a consistent manner? (Annotated bibliographies are not interesting)

Is the coverage balanced and thorough?

Is all the important literature cited?

Is the presentation biased, slanted, selective?

CMPUT 603 - Teaching and Research Methods

9

Other Important Issues

Simultaneous submissions, prior publication, and unrevised retries

Acknowledgments and plagiarismTimely response and returning a paperThe author’s reputationConfidentialityConflicts of interestThe editor’s roleThe program chair’s role