Some Thoughts On Peer Review In The Global Internet

Context

Narayanan Komerath

Daniel Guggenheim School of Aerospace Engineering Georgia Institute of Technology

Atlanta

IMETI 2010, Orlando, FL June 2010

http://farm4.static.flickr.com/3112/3157622458_b19d039158.jpg

http://knowledgecenter.unr.edu/help/using/peer.aspx

“Scholarly and professional journals feature articles written by researchers and practitioners in a particular subject area.”

“The authors often have particular specialties. Peer groups of researchers, scholars and professionals within a specific discipline are the audience for

scholarly literature.”

“Well-accepted indicator of quality scholarship.”

“Process by which an author's peers read a paper submitted for publication.”

“A number of recognized researchers in the field will evaluate a manuscript and

recommend its publication, revision, or rejection.”

“Articles accepted for publication through a peer review process implicitly meet

the discipline's expected standards of expertise.”

“Articles selected by an editor or board. Standards of scholarship in such journals

are often equal or comparable to those of peer-reviewed publications, although this is not always the case.”

“What is a scholarly or peer reviewed journal?”

“What is a Peer-Reviewed Journal?

..Sends articles submitted by authors to a group of experts in the field for review before publication. …The review process helps to ensure that research published

in the journal is of high quality and contributes new information to the field. Characteristics of Peer-Reviewed Journals:

Purpose is to share the results of original research with the rest of the

scholarly world. Usually have a plain cover with lengthy and in-depth articles that contain charts,

graphs, and tables. ..Minimal use of photographs and color. Any advertising is geared towards the subject discipline.

Articles are written by researchers, scientists or scholars in the field.

Authors cite their sources in footnotes or bibliographies, which are often extensive.

Uses the technical terminology appropriate to the discipline and assumes that the reader will have a similar scholarly background.”

From West Virginia University Library

http://undsci.berkeley.edu/images/us101/

peerreview.gif

http://www.whitehead.mit.edu/news/

paradigm/spring_2008/img/peer_review.jpg

http://1.bp.blogspot.com

Reviewer’s S.O.P. According to n.k.

1. Read the paper back and forth at least 3 times. Relate conclusions to what is

shown. Put away for a couple of days and mull on the paper. 2. Consider originality. Search for related prior work if necessary.

3. Follow author’s train of thought and check for reasonableness. 4. Read carefully and note questions

5. Compose review and pose questions

6. Develop summary recommendations and ratings 7. Get the review back to the editor in a timely manner.

8. If requested to review again, do so quickly.

A good peer review should engage the brains of the authors and the editors.

Takes time to do it right, but your professional standards are at stake.

How would you like someone like yourself as reviewer when you are the author?

My Background

32 years, full-time researcher/ academic tenure track/ tenured professor in a major

technological institution - Combustion diagnostics / propulsion experiments

- Turbulent combustion/ fluid dynamics - DSP / Optical and other diagnostics development

- Experimental aerodynamics / wind tunnel operations / vortex flows / combat aircraft /

helicopters - Guiding 15+ engineering PhDs.

- Institutional peer review panels. - Proposal and paper reviewer for national and international organizations

~ 250 publications, ~90 of them peer-reviewed. ~ 14 years internet / interdisciplinary / digital library / resource building & maintenance

~ 14 years in formally multidisciplinary endeavors ~ 12 years in Strategic / International / Security affairs

~ 8 years in social sciences interactions in community vs. academics

Prevailing Definition Questions/Criticisms

“Journals feature articles written by researchers and

practitioners in a particular subject area.”

•Narrow?

•Are credentials relevant? •Excuse for incompetent reviewing?

•Tactic to keep out “disruptive ideas”?

“The authors often have particular specialties. Peer groups

of researchers, scholars and professionals within a specific discipline are the audience for scholarly literature.”

Modify to accommodate cross-

disciplinary endeavor?

“Well-accepted indicator of quality scholarship.” Circular metric?

“Process by which author's peers read a paper submitted” Do they really? CAN they?

“A number of recognized researchers in the field will

evaluate .. and recommend publication, revision, or rejection.”

Are researchers scholars?

“Articles accepted for publication implicitly meet the

discipline's expected standards of expertise.”

Circular logic?

If “Articles are selected by an editor or board: Standards of

scholarship in such journals are ..not always comparable to those of peer-reviewed publications”

What is the absent process here? Are

Editors and the Board of a journal experts? Researchers? Scholars?

Some Questions From the Cross-disciplinary Perspective

The dual role of quality control and motivation encounters conflicts of

interest, especially when there are underlying competitive aspects and financial pressures.

•Delay

•Dismissive rejection, or glowing acceptance

•Unusually harsh or mild comments

•Demand for “additional work”

•Delay following rebuttal

•“Re-review” to think up new

excuses for rejection

“Killing” (or sneaking in) a Paper is Easy if the Editor is Asleep (or Worse)

http://2.bp.blogspot.com

So why have Peer Review?

Thoughtful review and comments from interested and knowledgeable persons is

more relevant than ever to motivating original thought, disciplined research and progress.

1. Help set in context of prior work

2. Help author improve thinking, calculation, presentation and writing: Ask questions that may not have occurred to author

3. Catch errors 4. Provide opportunity for author to rebut/revise/explain

5. Increase utility to other readers through an open discussion of

important issues.

What I have deleted is

“Accept only High-Quality Papers”

Observations on rejecting papers

1. Some process for rejection is essential, if only to encourage other authors.

2. High rejection rate is a low-quality metric of high quality.

3. Professional organization’s responsibility to not publish nonsense is outweighed by the risk

of rejecting innovative work.

4. There are better ways …..

•Reviewers are anonymous,

and do not have to defend their (in)actions

•Editors can delay publication

• Willing reviewers are hard to find

• Competent reviewers are harder to find

– and are usually competitors

Not possible in traditional model, to eliminate conflicts of interest.

Evolving technology and habits of researchers, and the attendant

fragmentation of time that many experts face, have driven traditional peer

reviewed paper journals to the verge of irrelevance.

•Adversarial / arrogant attitude (IP is

author’s!)

•Poor value addition by sloppy reviewing

•Cost (“paper charges”)

•Delay in review and publication

•Poor visibility: Libraries cannot afford to

subscribe to all, or most, journals

•Misses the Google Searcher Audience

• Citations will be more on papers that

are accessible!

Present generation does not follow

linear or hierarchial methods of knowledge organization

“If u can’t b found on Google screens 1-3, u don’t exist”

www.columbia.edu/cu/21stC/issue-1.1/peer.htm

Casual adoption of technology in review systems poses its own

problems, especially when used in reviewing innovative work

•“Review Panels typically support the status quo” (priceless quote from an

experienced authority from this conference!) •Adding technical convenience does not improve the amount or quality of time

and thought devoted to review.

http://francisthemulenews.files.wordpress.com/2009/04/dibujo20090425_peer_review_not_always_useful_for_editors.jpg

Academic promotion/tenure

processes are relying more on “prestige” and “percentile rankings

in a field” metrics along with number counts of publications

Fragmentation & proliferation of journals inhibits cross-disciplinary

innovation.

I reject this route as being

destructive of original thought and cross-disciplinary

knowledge dissemination.

http://imvu.regnarts.com/news/wp-content/

uploads/2008/06/peer-review-tears.png

Interim solution adopted by specialist communities to the lack of

competent “expert” reviewers in their narrow fields:

Creation of ever tighter “peer” circles, leading to fragmentation of knowledge and the inability of any library to maintain

a comprehensive journal base.

Thus the most “prestigious” publications may be so exclusive that few can even

find them.

“Professor Komerath is one of the Top 3 Experts in the World

in measuring pressure-velocity correlations in turbulent premixed flames using cross-spectral analysis of response-compensated Pitot microphone probes

with ceramic tips and a coherence metric”

(Great, but who else would want to join that club?)

Scholarly publication and academic freedom

vs. responsible community behavior

Egregious examples of abuses in “religion” academia.

Refusal of the “Academy” to publish well-informed rebuttals and exposures from

lay practitioners destroys academy’s credibility.

“Journals feature articles written by researchers and practitioners in a particular

subject area…Peer groups of researchers, scholars and professionals within a specific discipline are the audience for scholarly literature.”

Lesson: Scholarly communities that do not welcome and facilitate well-reasoned

disagreement from “lay persons”, are not worthy of respect

Technological capabilities and evolving trends in capturing

knowledge from prior work

•Internet Search Engines

•“Google Books” and “Google Scholar” •Delivery to personal communication devices (Iphone, Ipad)

•Global access • Moderated Discussion forum

• “Twitter”

Open, two-stage peer-review From Nature (2006) | doi 10.1038/nature04988

Stage 1: Rapid pre-screening (access review) and immediate publication as

“discussion paper” on website. Public interactive discussion for 8 weeks, with comments of designated reviewers & others are published alongside the

discussion paper. Reviewers sign or remain anonymous, but comments by other scientists must be signed.

Stage 2: traditional manuscript revision and peer review. If accepted, final papers are published in the main journal. Every discussion paper and

interactive comment remains permanently archived and individually citable.

Journal of Atmospheric Chemistry and Physics

Peer review model for cross-disciplinary innovation

Model Rationale

Web-based journal with periodical

volume and date identifiers

Archival, with swift global search

enabled

Submitted on-line in specified format For possible hardcopy publication

Assigned to 3 anonymous reviewers

by editors after editorial review.

Editor can reject. Review assignment

all over the world

Article posted on-line for comments

(BLOG model). Author can respond.

No censorship. Comments

moderated.

Anonymous review comments sent to

author only. Author has fixed period to modify or withdraw article

Encourages reviewers to help author

improve article without penalty

Where disagreement remains, both

the anonymous comment and the author’s rebuttal are published.

Main quality control: Archival reader

sees countering opinions – and author’s rebuttal

Article plus discussion becomes

archival record.

Captures discussion and allows

reader to form an informed opinion

Further comments published as new. Provides closure in a reasonable time.

Preliminary empirical observations

•Social-media forum to motivate experts to participate:

•Examples: 1. Strategic affairs discussion forum

2. Space Solar Power forum 3. Discussions on major news items (Gulf of Mexico oil leak)

•Most engineering journals are not likely to generate many comments from readers. •Getting authors to submit articles, requires recognition of quality control processes

and archival reliability of “journal” •Strong moderation is required

•Author ego is a problem with many authors unused to social

media

Notes: J. Atmospheric Science reports excellent impact factor.

“Deters submission of low-quality papers: low rejection rate”

Conclusions

• Purpose of peer review should be carefully re-emphasized

• Abuses cannot be avoided with traditional paper journal review system

• Swift & unobstructed publication essential.

• Open but moderated, interactive discussion provides good quality

• Archived reviews/rebuttal is a good experiment

• Motivating participation requires thought and resources

http://www.the-funneled-web.com/images_2008/PeerReview-ACS.gif

ACKNOWLEDGEMENTS

The work reported in this paper was made possible by resources being

developed for the “EXTROVERT” cross-disciplinary learning project under NASA Grant NNX09AF67G S01. Mr. Anthony Springer is the Technical Monitor.

http://

docinthemachine.com/

wordpress/wp-content/uploads/2007/02/

gspz0625.jpg