A Template for Producing Research Papers in the AI Lab Byron, Dan, Zan, and Jennifer Acknowledgement: Hundreds of meetings with Dr. Chen Disclaimer: Use

A Template for Producing Research Papers in the AI Lab

Byron, Dan, Zan, and JenniferAcknowledgement: Hundreds of meetings with Dr. Chen

Disclaimer: Use this advice at your own risk. If we already knew how to do it, it wouldn’t be research, and we would already have tenure!

Disclaimer 2: This presentation is filled with student perceptions of what Dr. Chen said or meant, and spiced with our own commentary. These views may or may not accurately reflect Dr. Chen’s position when you turn in a paper or give a presentation.

2

AgendaAgenda• Notes on the Title• Abstract• Introduction• Background and Literature Review• Research Questions• Research Testbed, System Design• Research Design• Experimental Results• Discussion• Conclusions and Future Directions• References• Some additional suggestions• Reviewing papers

3

Notes on the TitleNotes on the Title• A good title should reflect the entire contents

of the paper, avoid using cute titles.• Should be less than 8-9 words. • Complete sentences are not necessary, use

colons if needed.• Acronyms

– If needed, select meaningful acronyms that convey the meaning of the work/project (e.g., COPLINK, GeneScence).

– Acronyms help in branding a system/project.• Do not forget acknowledging the funding

agencies on title slides of presentations!

4

Abstract – What is in it?Abstract – What is in it?• One of the most important parts of the paper.• Concisely express the problem in one sentence or

two.• Mention why the work is important (if the goal of the

research was accomplished, what good thing would happen… )

• Describe methodology• Highlight the most important results

– Gives the reviewer a reason to continue reading the paper

• Should consist of short sentences– Don’t invent new words!

* Slides do not generally include an abstract *

5

Abstract – Our AbstractAbstract – Our Abstract• Publish, Publish, Publish (or Perish!)• Although many different methodologies can

lead to an accepted publication, these principles should be particularly useful to you in the AI Lab environment.

• We will present our view of how to get papers past Dr. Chen and out for publication.

• The content is presented in the form of a template.

6

Introduction – What is in it?Introduction – What is in it?• The Introduction “hooks” the reader.

– What is the motivation for the work?– What is the context?– Big picture, how did we address the problem?– What will the rest of the paper look like?– Introduction should be 3-5 paragraphs

• A four page introduction is not a good introduction

– The last paragraph always points to the structure of the paper

7

Introduction – Motivation/ContextIntroduction – Motivation/Context

• PhD students want to learn to be productive researchers.

• A number of research approaches are effectively employed by various researchers and research groups.

• The model used here in the AI Lab has a strong track record of producing work publishable in top tier journals.

8

Introduction – The Big PictureIntroduction – The Big Picture

• To be productive we need to be:– Doing the right research,– Doing the research right, and– Packaging the research appropriately for

distribution.

• Missing any of these elements substantially reduces the chance that your work will be published.

9

Doing the Right ResearchDoing the Right Research• Criteria for choosing a research project that

might eventually be funded.– Does it advance Science? Choose work that

makes a contribution to some scientific body of knowledge. Implementing an effective system is not enough: the methodology and techniques are important.

– Does it have potential impact? Do work that will improve important real-world processes. Thus, we emphasize domain-specific applications and completing a line of work in a user study. Caveat: Making an impact with one paper is tough.

80%

20%

10

Doing the Research RightDoing the Research Right• Research should be publishable.• Strong methodology is vital. Are the

experiments rigorous and valid?– Precise hypotheses – Ideally, hypotheses are based on previous

literature or established theories– Appropriate statistical tests

• Even if the contribution is small, good methodology can get a paper over the top.– Methodological flaws give reviewers an excuse to

reject your work.

11

Right Research for Junior FacultyRight Research for Junior Faculty

• The should be publishable, not necessarily suited for funding.

• Extend your current work, don’t go down an entirely different path. Choose wisely in your Ph.D.

• Use your methodology on other collections and different contexts.

• Define an area, so people know you for your work.

12

Appropriate PackagingAppropriate Packaging

• Even solid work will be rejected if it is not appropriately packaged.

• Main ideas of good packaging:– Be concise– Be professional– Target the journal or conference– Be persuasive

13

Appropriate PackagingAppropriate Packaging• Good Slides

• Force you to organize concisely and clearly,• Allow Dr. Chen to present the work to keep the $$$

coming, and• Reduce the time needed to write a good paper.

• Good writing• Describes previous work in a digested form• Does not distract the reader• Makes a coherent argument• Employs good examples to illustrate difficult techniques or

concepts

• The transition from good slides to a good paper is 2-3 weeks.

14

Introduction – Looking AheadIntroduction – Looking Ahead

• In future sections we will:– Review the main points: Lit Review– Present our Research Questions– Describe important environmental issues:

Research Testbed– Discuss methodology: Research Design– Present our Results– Discuss the implications– Draw Conclusions and look to Future

Work

15

AgendaAgenda• Abstract• Introduction• Background and Literature Review• Research Questions• Research Testbed, System Design• Research Design• Research Findings• Discussion• Conclusions and Future Directions• Additional Suggestions• Reviewing Papers

16

Literature Review: What is it?Literature Review: What is it?• One of the most important parts of a paper• The lit review frames the work.• Connection between introduction and

research questions– Introduction points out the motivation– Literature review provides more evidence of the

limitations in previous studies– Following this logic flow, lit review leads to

specific research questions• What to do? e.g., new algorithm, performance, etc.

Although, a paper is sometimes accepted largely because of a

strong lit review that summarizes and organizes an area of inquiry

17

Literature Review: Key IdeaLiterature Review: Key Idea• The literature review presents “digested

material”– Taxonomies/Frameworks are good

A taxonomy of 2~4 dimensions

– Know all studies in the field and focus on relevant ones

– Tell what previous work means– Choose the right/relevant subset of all the papers

you could cite• Don’t try to review everything, understand the audience

of the paper

18

Literature Review: CompletenessLiterature Review: Completeness

• Different level of completeness depending on journal/audience– In general, a more comprehensive review– For special issues, not too big, more focused

• Different focus depending on your research question– To propose a new task,– To compare performance, or– Etc.

19

Literature Review: TipsLiterature Review: Tips• NOT*:

– “use too much tutorial,”– “educating the reader,” – “lose the seminal works,”– “making sure we mentioned everything” (No laundry lists!), or– “too critical to others’ work” (Maybe he/she is the reviewer)

• INSTEAD:– Enough coverage– Be Comprehensive– Critiques (what are missing leads to your research questions)– Show why our approach makes sense– Provide a benchmark for comparing our results

* Although, a paper is sometimes accepted largely because of a

strong lit review that summarizes and organizes an area of inquiry

20


21

Research QuestionsResearch Questions

• The intro said why. The lit review set up the argument.

• Research Questions:– focus the work suggesting what we can

measure,– follow logically from the lit review (address

the critics in lit review and lead to your findings), and

– should be answered by the experiment(s).

22

• Have 2-5 major research questions,

• They should have clear scientific motivations:– Innovation to basic science, and – potential impacts.

Research QuestionsResearch Questions

23


24

Research TestbedResearch Testbed

• What data sets will be used in the experiment(s)?

• Testbed should be interesting, relevant, and significant.

• We have:– Slides available from previous

presentations – Published papers

25

System DesignSystem Design• Describe how the architecture works and its

components• The basic publication flow:

Topic Identification

Prepare Initial Slides

Experimentation

Final Slides

Write a Paper

Write Grant Proposals

Format and Submit

Revise and Write Revision Letters

PresentationsConferences &

Funding Agencies

26

System DesignSystem Design

• Good diagrams help readers understand better and clarify our own thoughts.

• Behavioral papers may have a methodology section instead of system design.

• Algorithm papers discuss methodology and algorithm design (pseudo codes and diagrams are suggested) in this section.– Methodology needs to have a theoretical

foundation.

27


28

Research Design: What’s in it?Research Design: What’s in it?

• Focus on the experiment (s).• Present hypotheses

– Measurable– Address the research questions– Plan for statistical tests

29

Research Design: What’s in it?Research Design: What’s in it?

• Is to validate your research.• Use credible experiments to verify the

hypotheses.• Methodology:

– Quantitative measures: such as accuracy and speed.

– Qualitative measures: explains the inside phenomena of the quantitative results.

– Simulation: is often used in system design arena.

30


31

Research FindingsResearch Findings

• Tables and figures are critical.– Need to be consistent and neat.– Highlight interesting numbers.– In caption, you may use 3-4 sentences to

describe more details about a figure or a table.– Use a small paragraph in text to explain the

essence about a figure or a table.

• You may group your findings in chunks, each of which starts with a bold summarizing sentence.

32


33

DiscussionDiscussion

• The discussion section gives meaning to the results.– Why did you get the results you got?– If some of the results were surprising,

why?– What did you observe outside the

measured information presented in the research findings section?

34

Conclusions and Future DirectionsConclusions and Future Directions

• Can have some duplication with the abstract.

• State the contribution, but don’t overstate it. Don’t form questions in the reviewers mind.

• Don’t mention trivial future directions.• Point to several promising directions.

35

ReferencesReferences• Where has similar work been published?• What kind of articles are accepted by the target

journal?• Remember who did previous work.• Know where it was published.• Try to reference related papers that were published

in the target journal.• Must have 5-10 key journals, key conferences in the

field• Number the references• Have a consistent format

36


37

ProfessionalismProfessionalism

• Eliminate typos and grammar errors

• Consistent formatting

• Clear figures and tables– Captions make the meaning of the figure

clear– The layout should be clear and clean– Every figure/table must be referenced in

the text

38

PresentationsPresentations

• Control your time; 40 slides not 75!• Present with energy and enthusiasm• Listen to questions; you can clarify

before you answer• Don’t avoid questions, especially if they

ask for specific information.• Rehearse; know what slides come next• Don’t read from your slides

39

Be concise. Consider the flowBe concise. Consider the flow

• Be concise:– Do you need this

slide/sentence/word/paragraph?– Is the prose wordy?– Active sentences are better than passive.

• Flow: Why is this point here?– Present info in a logical, top-down flow– Good: Therefore…..– Bad: You’ll see why later….

40

An example of good flow (1 of 2)An example of good flow (1 of 2)

• What is the problem?• Why do we care?• How has it been addressed before?• What is the research gap?• How are you going to address the

research gap? – It should be clear from previous material or

input here why you chose each part of your solution.

41

An example of good flow (2 of 2)An example of good flow (2 of 2)

• How will you measure the results?• What were your results?• Were they statistically significant?• What did you learn?• Why is that important?• What will you do next?

42

More things to think aboutMore things to think about

• If you vary from the template you should have a good reason.

• Dr. Chen resists incomplete slides.• Multiple revisions improve your slides.

Get through a couple of revisions before you show them to Dr. Chen.

• Let your colleagues help.

43

A challenge:A challenge:

• Be ready with a good set of slides 3 days ahead.

• Practice presenting them.• Tighten and revise.• Have a final version no less than 24

hours in advance.• Practice presenting the slides!

44


45

• Be professional– Pretend everyone will see your review

• Be accurate

• Be specific

• Be critical– Of the methodology

Reviewing PapersReviewing Papers

46

• Read similar papers

• Summarize the paper

• Separate major and minor comments

– Your review becomes your reputation– Don’t rewrite the article

Reviewing PapersReviewing Papers

47

• 1-3 pages (never less than ½)– Include high level…

• Summary• Strengths• Weaknesses

• Selection Categories:– Accept as is

• NEVER choose this one• Indicates laziness (Yours!)

– Minor revisions• No methodology problems• Findings are interesting

Journal ReviewsJournal Reviews

48

• Selection categories cont…– Major revisions

• Paper can be fixed through a new experiment• Needs significant clarification

– Is incomplete

– Rejection• Wrong methodology/implementation• Findings are trivial/uninteresting

– Have I learned something new?

– Wrong Journal• Suggest a different journal• Not substantial enough, recommend as a short note

• Complete Journal Reviews in 2-3 months

Journal ReviewsJournal Reviews

49

• ½-1 page

• Accept only if light editing is necessary– No time for major overhauls

• Is research and methodology interesting?

• Complete conference reviews in 2-4 weeks

Conference ReviewConference Review

Documents

A Template for Producing Research Papers in the AI Lab Byron, Dan, Zan, and Jennifer Acknowledgement: Hundreds of meetings with Dr. Chen Disclaimer: Use