Research Seminar – CPIT 695

Reading Skills

1

What is required from you?

how research papers are written, how to read such papers critically and efficiently, how to summarise and review them. how to gain an understanding of a new field, in the

absence of a textbook how to judge the value of different contributions how to identify promising new directions

What is a Paper?

A paper: the prime medium for reporting scientific results In computer science: the conjunction of

something conceptual (and preferably new), something sound, and something practical that is relevant today (read: IT).

Reading Skills

In principle, a paper provides enough information for its reader to reproduce its contents: proof, experiment.

The three stages of reading (as one grows up): 1. the books say blah and thus it is true; 2. this book says blah; 3. the author wrote blah.

On reading critically Don’t swallow the author’s propaganda, accepting the paper

as the author shaped it: disassemble it to identify its real thrust; appreciate; probe / question / stress; (if needed) reassemble: minimize / expand.

Reading Skills

What if you don’t understand something? What if you think you found a bug? Recommendation:

1. Consult people locally. 2. Send a very polite e-mail to the author (keeping in mind

that you may well be wrong yourself). Prudence: You don’t just represent yourself. You represent your adviser and your institution.

Some elements for a review

1. Convey your understanding of the paper with a summary. 2. Double up with an analysis. 3. Sum up with an assessment and a recommendation. 4. Add a list of remarks, if any. Context of the review: Be objective. Be fair. Don’t delay. Be

courteous. Remain confidential. To summarize

Reviews should be as comprehensive as possible. Reviews should be as courteous as possible. Reviews should be as selfless as possible.

Reading Skills

Don't let other people's frame of mind limit your creativity. Proofreading each other

What: an investment. How: with consideration. Be prepared to be misunderstood.

Why read papers So you know what’s happening Avoid reinventing the wheel

does happen commonly, too many wheels already Find interesting research topics

Why not to read papers Cannot read everything Should not read everything Can suppress innovation: once you see solutions using a particular theme,

often hard to think differently Pitfall

It is of no use to spend almost all of your time in literature review and seminars.

It is impossible to ``finish a literature review and then start research." New literature is always appearing, and as your depth and breadth

increases, you will continually see new connections and related areas that must be studied.

Active listening and reading must be viewed as ``continuing education'' that will involve you for the rest of your career.

Don't fool yourself into thinking it must be finished before you can begin research.

Read or not to read, that is the question

Read, of course Know what’s important Know what can be ignored without significant loss of

information

What’s in a paper

Abstract Introduction Motivation Problem description Solution ... Performance Analysis Conclusions Future Work

How to read a paper?

Know why you want to read the paper To know what’s going on (e.g., scanning

proceedings) title, authors, abstract

Papers in your broad research area introduction, motivation, solution description, summary,

conclusions sometimes reading more details useful, but not always

Papers you may want to improve on read entire paper carefully

What to note

Authors and research group Need to know where to look for a paper on particular topic

Theme of the solution Should be able to go back to the paper if you need more info

Approach to performance evaluation

Note any shortcomings

Reading Skills To develop an effective reading style for research papers, it can help to know

what you should get out of the paper, and where that information is located in the paper.

Typically, the introduction will state not only the motivations behind the work, but also outline the solution. Often this may be all the expert requires from the paper.

The body of the paper states the authors' solution to the problem in detail, and should also describe a detailed evaluation of the solution in terms of arguments or an empirical evaluation (case study, experiment, etc.).

Finally, the paper will conclude with a recap, including a discussion of the primary contributions.

A paper will also discuss related work to some degree. Papers are often repetitive because they present information at different levels

of detail and from different perspectives. As a result, it may be desirable, to read the paper ``out of order'' or to skip

certain sections.

Questions to be answered while reading a paper

What are motivations for this work? For a research paper, there is an expectation that a solution to a problem has been found that no one else has published in the literature. The paper should describe why the problem is important and why it does not have a trivial solution; that is, why a new solution may be required. Implicitly there is implication that previous solutions to the problem are inadequate. Occasionally an author will fail to state either point, making your job much more difficult.

What is the proposed solution? This is also called the hypothesis or idea. There should also be an argument about why the solution solves the problem better than previous solutions. There should also be a discussion about how the solution is achieved (designed and implemented) or is at least achievable. Are all concepts and notations introduced before their first usage?

Questions to be answered while reading a paper

What is the evaluation of the proposed solution? An idea alone is usually not adequate for publication of a research paper. What argument and/or experiment is made to make a case for the value of the ideas? What benefits or problems are identified? Are they convincing?

What alternative solutions exist? Read a paper critically. What is your analysis of the identified problem, idea and

evaluation? Is this a good idea? What flaws do you perceive in the work? What are the most interesting points made? What are the most controversial ideas or points made? For work that has practical implications, you also want to ask: Is this really going to work, who would want it, what it will take to give it to them, and when might it become a reality?

Questions to be answered while reading a paper

What are the contributions? How does this contribution relate to work previously encountered? The contributions in a paper may be many and varied. Ideas, software, experimental techniques, and area survey are a few key possibilities.

What are future directions for this research? Not only what future directions do the authors identify, but what ideas did you come up with while reading the paper? Sometimes these may be identified as shortcomings or other critiques in the current work.

What questions are you left with? What questions would you like to raise in an open discussion of the work? What do you find confusing or difficult to understand? By taking the time to list several, you will be forced to think more deeply about the work.

What is your take-away message from this paper? Sum up the main implication of the paper from your perspective. This is useful for very quick review and refreshing your memory. It also forces you to try to identify the essence of the work.

What are the important references cited in this paper by the author?

Reading Skills

Presumably, the introduction should provide motivation. The introduction and conclusion may discuss the solutions and

evaluation at a high level. Future work is likely in the concluding part of the paper. The details of the solution and the evaluation should be in the

body of the paper. You may find it productive to try to answer each question in turn,

writing your answer down. In practice, you are not done reading a paper until you can

answer all the questions. Use the form (PaperReviewForm.pdf) given to fill OR write

an abstract using the answers from the questions.

Getting the bigger picture

What problem is the paper trying to solve? Highlight three to four lines

What are the limitations of prior work? Highlight maximum two to three lines

How is the problem solved by this paper? What is the paper s’ contribution? Highlight three to five lines

How is the proposed solution evaluated? What kind of data/experiments were conducted? No need to highlight anything; you can highlight two to three

words here

Getting the Details What problem is the paper trying to solve? Read the System Model carefully

The whole paper is going to be based on this model Understand the gist of the contribution/proposal

Does it make sense? Do you think it will work? How will the proposal be evaluated?

Don’t try to read all the math in one go Read the assumptions and system model Try to work out a solution to the problem

Is the evaluation fair and comprehensive? Try to find the next paper you want to read from this section

Understand the results Do not miss a single figure and table Find the corresponding discussions in the paper and read them thoroughly

Reading Skills – What to retain?

The problem The basic idea of the proposed solution Your personal notes on the paper’s mathematics Shortcomings of the proposed approach

Reading Skills – Ask yourself

What is the message you take away from this paper? Are you convinced that the paper attempted an important

problem? If your answer is NO, Justify it! If your answer is NO, Justify it!

‰Are you convinced that the paper proposed a viable solution? If your answer is NO, Justify it!

‰What questions still remain unanswered?

If you find a well-written paper

After reading a well written paper, consider the presentation. It may be nicely organized, or it may underscore concepts with thoughtful examples. What makes this paper easy to read? What level of detail is provided? What examples are used to demonstrate important concepts? What questions are left unanswered? Can the results be generalized?

Getting most out of what you read Be organized

Keep an electronic bibliography with notes & pointers to the paper files Keep and file all the papers you have read or skimmed

Be efficient - only read what you need to Start by reading only the conclusion, scanning figures & tables, and

looking at their references Read the other sections only if the paper seems relevant or you think it

may help you get a different perspective Skip the sections that you already understand (often the background and

motivation sections) Take notes on every paper you find worth reading

What problem are they trying to solve? What is their approach? How is it different from other approaches?

Getting most out of what you read Summarize what you have read on each topic - after you have read several

papers covering some topic, note the: key problems various formulations of the problem they are addressing relationship among the various approaches alternative approaches

Read PhD theses - even though they are long they can be very helpful in quickly learning about what has been done is some field. Especially focus on:

Background sections Method sections Your advisor's thesis - this will give you an idea for what he/she expects

from you.

Summary in matrix or table form

References CPIT 695 Notes of Dr. Ahmad Barnawi Materials to refer:

How to read a CS research paper? – Philip Paper Review Form

http://www.cs.brandeis.edu/~cs227b/papers/introduction/howToRead.txt http://wisnet.seecs.nust.edu.pk/people/~khayam/pdf/

lecture_research_paper_reading.pdf http://cseweb.ucsd.edu/~wgg/CSE210/howtoread.html http://www2.cs.uregina.ca/~pwlfong/CS499/reading-paper.pdf http://cs.au.dk/~danvy/tips-and-tricks/index.html http://faculty.washington.edu/wpratt/survive.htm http://www.michbar.org/generalinfo/plainenglish/ “How to have Your Abstract rejected” Mary-Claire van Leunen and Richard

Lipton “Maxims for Malfeasant Speakers” Norman Ramsey 26