Alternative Peer Review : Quality Management for 21 st Century Scholarship

Alternative Peer Review:Quality Management

for 21st Century Scholarship

Gerry McKiernan

Science and Technology Librarian

and Bibliographer

Iowa State University Library

Ames IA

[email protected]

http://www.public.iastate.edu/~gerrymck/APR.ppt

Workshop on Peer Review in the Age of Open Archives

International School for Advanced StudiesInterdisciplinary Laboratory

Trieste, ItalyMay 23-24, 2003

THANK YOU!Workshop Advisory Board [(Marco Fabbrichesi (INFN/SISSA Italy), Stevan Harnad (University of Southampton, UK), Stefano Mizzaro (University of Udine, Italy) and Corrado Pettenati (CERN Library, Geneva, Switzerland)]Iowa State University, Faculty Senate, Committee on Recognition and DevelopmentEuropean CommissionIowa State University LibraryHeike Kross, Ph.D.

DISCLAIMER (1)The screen prints selected for this presentation are for educational

purposes and their inclusion does not constitute an endorsement of an associated product, service,

place, or institution.

DISCLAIMER (2)The views and opinions expressed

in this presentation are those of the presenter and do not constitute

an endorsement by Iowa State University or its Library.

NOTICE

No editors, authors, or referees were harmed in the preparation of

this presentation.

http://www.sciencecartoonsplus.com/

Giuseppe De Nittis (1846-1884)The Macchiaioli / Italian Impressionists

Campo di Biche (1875)

PEER REVIEW: DEFINITION

“Peer review is the assessment by an expert of material submitted for publication.”

Carin M. Olson, “Peer Review of the Biomedical Literature,” American Journal of Emergency Medicine 8 no.4 (July 1990): 356-358.

PEER REVIEW: PURPOSES


Peer review helps to ensure that published research is:

Important Original

Timely Technically-reliable

Internally-consistent Well-presented

Benefited from guidance by experts

PEER REVIEW: STRENGTHS

Anne C. Weller, Editorial Peer Review: Its Strengths and Weaknesses. (Medford, NJ: Information Today, 2001).

The underlying strength of peer review is“…the concerted effort by large numbers of researchers and scholars who work to assure that valid and valuable works are published and conversely to assure that invalid or non-valuable works are not published … .”

“Houston, We Have a Problem!”

PEER REVIEW: PROBLEMSSubjectivity

Bias

Abuse

Detecting defects

Fraud and Misconduct

DelayFytton Rowland, “The Peer-Review Process,”

Learned Publishing 15 no. 4 (October 2002): 247-258.

Report version: http://www.jisc.ac.uk/uploaded_documents/rowland.pdf

SUBJECTIVITYSummary rejections by editor without sending the paper to referees

Choice of referees by the editor (choosing for example, a known harsh referee for a paper the editor wishes to see rejected)

BIASDiscrimination against authors because of their nationality, native language, gender or host institution

Situations where author and referee are competitors in some sense, or belong to warring schools of thought

ABUSEToo many articles out of one piece of research, or duplicate publicationIntellectual theft: omission or downgrading of junior staff by senior authorsPlagiarism (stealing others yet unpublished work that has been sent for review)Delaying publication of potentially competing research

DETECTING DEFECTSIdentification of factual errors within submission

FRAUD and MISCONDUCTFabrication of results

Falsification of data

False claim of authorship for results

DELAY“There is much muttering about publication delay, a real enough problem, especially in paper publication, but peer review itself is often responsible for as much of the delay as the paper publication and distribution process itself.”

Stevan Harnad

Stevan Harnad, “Implementing Peer Review on the Net: Scientific Quality Control in Scholarly Electronic Journals, in Scholarly

Publication: The Electronic Frontier, edited by Robin P. Peek and Gregory B. Newby (Cambridge MA: MIT Press, 1996).

http://www.ecs.soton.ac.uk/~harnad/Papers/Harnad/harnad96.peer.review.html

http://www-marketing.wharton.upenn.edu/ideas/pdf/peerpaper.pdf

“Peer review is slow, expensive, profligate of academic time, highly subjective, prone to bias,

easily abused, poor at detecting gross defects, and almost useless in detecting fraud.”

Richard Smith, “Opening Up BMJ Peer Review,”

BMJ 318 (7175) (January 2 1999): 4-5

Richard SmithEditor, BMJ

Stephen Lock, A Difficult Balance: Editorial Peer Review in Medicine (Philadelphia, PA: ISI Press, 1986).

EXAMPLE

Jan Hendrik Schön

http://archive.salon.com/tech/feature/2002/09/16/physics/

-

Science

Nature

http://physicsweb.org/article/news/6/9/15/

http://agenda.cern.ch/fullAgenda.php?ida=a01193

RECOMMENDATIONSWorkshop on the Open Archives Initiative (OAI)

and Peer Review Journals in Europe, CERN, Geneva Switzerland, March 22-24, 2001

“The participants were unanimous in their belief that the certification of scholarly work remains a fundamental part

of a system for scholarly communication.”

“It was [also] generally believed that the electronic environment allows for novel approaches to accord a

stamp of quality to scholarly works.”

Alison Buckholtz, Raf Dekeyser, Melissa Hagemann, Thomas Krichel, and Herbert Van de Sompel, “Open Access: Restoring Scientific Communication to Its Rightful Owners,” European Science Foundation Policy Briefing 21 (April 2003): 1-8.

http://www.arl.org/sparc/SPB21_OAI.pdf

“Let us be imaginative in exploring the remarkable possibilities of this

brave new medium.”

Stevan Harnad, “Implementing Peer Review on the Net: Scientific Quality Control in Scholarly Electronic Journals, in Scholarly Publication: The Electronic

Frontier, edited by Robin P. Peek and Gregory B. Newby (Cambridge MA: MIT Press, 1996).

http://www.ecs.soton.ac.uk/~harnad/Papers/Harnad/harnad96.peer.review.html

“Let us be more imaginative in exploring the remarkable possibilities of this

brave new medium.” With Apologies to Stevan Harnad

http://lockss.stanford.edu/

TM

LOCKSSFor centuries libraries and publishers have had stable roles: publishers produced information; libraries kept it safe for reader access. There is no fundamental reason for the online environment to force institutions to abandon these roles.

The LOCKSS model capitalizes on the traditional roles of libraries and publishers. LOCKSS creates low-cost, persistent digital "caches" of authoritative versions of http-delivered content.

LOCKSSThe LOCKSS software enables institutions to locally collect, store, preserve, and archive authorized content thus safeguarding their community's access to that content.

The LOCKSS model enforces the publisher's access control systems and, for many publishers, does no harm to their business models.

LAMPSS

Lots of Alternative Models Provide Sensible Solutions

DISCLAIMER

The alternative peer review models profiled are for informational and

educational purposes only and do not

necessarily constitute an endorsement.

ON

ALTERNATIVE PEER REVIEWNeo-ClassicalCertification-basedOpen Peer ReviewCommentary-basedCollaborately-filtered

Institution-basedCitation-basedIndex-basedMetadata-basedComputer-assistedNO Peer ReviewModerator-basedTier-based

NEO-CLASSICAL PEER REVIEW

Neo-Classical Peer Review



CERTIFICATION-BASED

Certification-Based“The process of pre-publication peer

review could be improved and become a more reliable indicator of manuscript quality if reviewers were trained in, and subsequently applied systematically, critical skills and use of a hierarchy of evidence to classify submitted articles being reviewed.”

Stephen Pritchard , “Peer Review - a Proposal for Change,”Paper presented at Thinking Globally - Acting Locally:

Medical Libraries at the Turn of an Era,8th European Conference of Health and Medical Libraries,

September 16-21, 2002, Deutsche Zentralbibliothek für Medizin, Köln, Germany.http://www.zbmed.de/eahil2002/abstracts/pritchard.pdf

OPEN PEER REVIEW

Open Peer ReviewIDENTIFICATION OF REVIEWERS / SIGNED REVIEWS

BMJ bmj.com

BioMed Centralbiomedcentral.com

eMJA (Medical Journal of Australia)www.mja.com.au/public/information/project.html

COMMENTARY-BASED

Commentary-basedReaders can comment before and/or after classic peer

review, or instead of classic peer review

Electronic Transactions on Artificial Intelligence (www.etaij.org)

OPEN REVIEW / REFEERING

Journal of Interactive Media in Education (www-jime.open.ac.uk)

PRE- AND POST- COMMENTARY

Psycoloquy

(psycprints.ecs.soton.ac.uk)

POST PEER REVIEW COMMENTARY)

Electronic Transactions on Artificial Intelligence

Journal of Interactive Media in Education

http://psycprints.ecs.soton.ac.uk/

COLLABORATIVELY-FILTERED

Collaboratively-Filtered

“DEFINITION: “Guiding people's choices of what to read, what to look at, what to watch, what to listen to (the filtering part); and doing that guidance based on information gathered from some other people (the collaborative part)."

Paul Resnick

http://www.cni.org/Hforums/cni-announce/1996/0031.html

ResearchIndex / CiteSeer

http://www.researchindex.com

INSTITUTION-BASED

Institution-BasedINSTITUTIONAL INITIATIVES

DSpace™ (MIT)www.dspace.org

eScholarship (University of California)escholarship.cdlib.org

Glasgow ePrint Service (University of Glasgow)eprints.lib.gla.ac.uk

CITATION-BASED

Citation-Based Citations to Open Access / OAI-compliant

documents are indicators of document importance

http://citebase.eprints.org/

INDEX-BASED

Index-Based INDEXING OF EPRINTS BY COMMERICAL ABSTRACTING AND

INDEXING SERVICE

Chemical Abstracts (American Chemical Society) (CAS) indexes select appropriate e-prints from the arXiv.org eprint server as well from the Chemical Preprint Service (Elsevier)Its “selection criteria for this kind of electronic document are essentially the same as for the traditional printed documents: they must report new information of chemical or chemistry-related interest and must be original publications. Also, the electronic publication must be publicly available and have some relative permanence ….”

Eric Shively / Chemical Abstracts Service

Index-Based “CAB Abstracts doesn’t currently include Eprints or Preprints, but we are looking at the implications and possible mechanisms for accessing and indexing Eprints and/or Preprints related to the applied life sciences.”

Tracy Shaw / CAB International

METADATA-BASED

Metadata-Based<oai-quality>

<category>internal</category>

<process>

peer review

</process>

<organization>

CERN

</organization>

<policies>

http://www.cern.ch/policies/review.html

</policies>

</oai-quality>

William Y. Arms, “Quality Control in Scholarly Publishing On The Web. What Are the Alternatives to Peer Review?” PowerPoint presentation given at the Workshop on the Open

Archives Initiatives (OAI) and Peer Review Journals in Europe, March 22-24, 2003, CERN, Geneva, Switzerland.http://www.cs.cornell.edu/wya/papers/CERN-2001.ppt

COMPUTER-ASSISTED

Computer-Assisted (1)SOFTWARE THAT ASSISTS IN THE EVALUATION OF A

SUBMITTED MANUSCRIPTA Software Program to Aid in Peer ReviewAlvar Loria and Gladys Faba

Objective: To characterize a personal computer-based software program developed as an aid to peer review of medical papers. Design: The software is a Windows-based application that records automatically a numeric score to a series of questions related to 8 sections of scientific papers (introduction, methods, results, and discussion, plus 4 other sections). The questions and sections vary according to type of paper (original reports, case reports, or reviews), and the final output is a score with a maximum of 100 for a "perfect" paper. The software was tested using a single reviewer to judge 289 papers (169 original reports, 50 case reports, and 70 reviews) from 44 Mexican medical journals. All statistical analysis of scores were done with nonparametric tests.

Computer-Assisted (2)Results: The paper scores ranged from 29 to 97 with slightly higher median and less dispersion of scores for reviews as compared with original reports and case reports, but these differences did not reach significance. Two observations suggest that the software operated reasonably well: a) there were some differences in the section scores by type of paper that agreed well with differences in their complexity; b) the journal scores showed an association with their number of original papers and their percentage of original papers (Kruskal-Wallis test, P=.06 and 0.07, respectively). Conclusions: The software operated reasonably well when used to compare the relative quality of 289 papers. The validity of the program is restricted in this study to the experience of 1 reviewer. An analysis of the raw scores helped in detecting some ambiguous and redundant questions that have been modified in an improved version. The program has a potential as a training tool for inexperienced reviewers or as a scorekeeper for experienced peer reviewers.

Alvar Loria and Gladys Faba, “A Software Program to Aid in Peer Review,” Abstract of paper presented at the Third International Congress on Biomedical Peer Review and Global Communications ,

September 18-20, 1997, Prague, Czech Republic. http://www.ama-assn.org/public/peer/arev.htm

NO PEER REVIEW

NO Peer Review

http://xxx.arXiv.cornell.edu

MODERATOR-BASED

Moderator-based (1)The intent of this model is to allow the widest range of scientific manuscripts to be archived, searched, and distributed electronically at the lowest possible cost.This would be accomplished through very minimal filtering and subsequent placement of eprints on a non-commercial archival server by a subject-specific Moderator appointed by a society (or consortia of societies).A society-appointed Editorial Board (with double-blind peer review approved by the non-profit Peer Review Inc. organization) would then the identify the most important materials from among these archived items, and the stamp of approval for these items would be included in a secondary Virtual Collection.

Moderator-based (2)There are no direct submissions to the Editorial Board; manuscripts would be directed to the Editorial Board in one of three ways:

1. nominated by the eprint Moderator upon receipt for the archival server,

2. notification sent to the Editorial Board when a threshold number of hits are generated by any one manuscript on the archive server, and

3. nominated by readers of material from the archive; this process requires a letter of support outlining the importance of the work to the Editorial Board.

Moderator-based (3)The Virtual Collection could be produced as a variety of products:

enhanced abstracts email threads (with comments) virtual reviews of sub-disciplines SDIs (selective dissemination of information) current awareness tools

This process: reduces the load on the Editorial Boards, which results in a faster review process; differentiates those items worthy of higher recognition from those worthy of archiving, making it easier for a reader to filter material, based upon a society and discipline authority (rather than commercial reasoning);provides for search/browse/sdi from the Virtual Collection for filtered info, reducing this more expensive option for only those items recognized as of the highest quality.

David Stern, “The eprint Moderator Model,” Newsletter on Serials Pricing Issues no. 214 (February 8, 1999).

http://www.lib.unc.edu/prices/1999/PRIC214.HTML#214.5

1

2 3

TIER-BASED

Tier-basedTwo separate domainsStandard Tier Any and all submissions would be accepted after a cursory examination of or other pro forma certification. The review process could be “minimally labor-intensive, perhaps relying primarily on an automated check of author institutional affiliation, prior publication record, research grant status, or other related background; and involve human labor primarily to adjudicate incomplete or ambiguous results of an automated pass.”

Tier-basedUpper Tier “At some later point (which could vary from article to article, perhaps with no time limit), a much smaller set of articles would be selected for the full peer review process. The initial selection criteria for this smaller set could be any of a variety of impact measures, to be determined, and based explicitly on their prior widespread and systematic availability and citability: e.g., reader nomination or rating, citation impact, usage statistics, editorial selection, ... .”

Paul Ginsparg, “Can Peer Review be Better Focused?,” Science & Technology Libraries 22 No. 3/4 (In press).

http://arxiv.org/blurb/pg02pr.html

DISCLAIMER

Presented for Your Consideration

OFF

FREEDOM OF IDEAS

http://www.nrm.org/exhibits/current/four-freedoms.html

‘FUTURE OF IDEAS’ The explosion of innovation we have seen

in the environment of the Internet was not conjured from some new, previously

unimagined technological magic; instead, it came from an ideal as old as the nation. Creativity flourished there because the

Internet protected an innovation commons.

‘FUTURE OF IDEAS’

The Internet’s very design built a neutral platform upon which the widest range of

creators could experiment.

The legal architecture surrounding it protected this free space so that culture and information –

the ideas of our era–could flow freely and inspire an unprecedented breadth of expression.

Lawrence Lessig, The Future of Ideas: The Fate of the Commons in a Connected World

. (New York: Random House, c2001).

“IT’S

NOT

ABOUT

PUBLICATION;

IT’S

ABOUT

IDEAS.”

The Association of Learned and Professional Society Publishers (ALPSP) Survey

Authors and Electronic PublishingScholarly research communication has seen far-reaching developments in recent years. Most journals are now available online as well as in print, and numerous electronic-only journals have been launched; the Internet opens up new ways for journals to operate. Authors have also become conscious of alternative ways to communicate their findings, and much has been written about what they ought to think.

ALPSP felt that it would be timely to discover what they actually thought and what they actually did. This survey aimed to discover the views of academics, both as authors and as readers. Some 14,000 scholars were contacted across all disciplines and all parts of the world, and nearly 9% responded; their detailed comments make thought-provoking reading.

Alma Swan and Sheridan Brown. Authors and Electronic Publishing: The ALPSP Research Study on Authors' and Readers’ Views of Electronic Research Communication. (West Sussex, UK: The Association of Learned

and Professional Society Publishers, 2002).http://www.alpsp.org/pub5.htm

“When asked to predict what would be the most common form of quality

control in five years time, only a bare majority answered

‘traditional peer review’.”

Fytton Rowland, “The Peer-Review Process,” Learned Publishing 15 no. 4 (October 2002): 247-258.


FURTHERMORE …16% said that the referees would no longer be anonymous27% said that traditional peer review would be supplemented by post-publication commentary 45% expected to see some changes in the peer-review system within the next five years

Fytton Rowland, “The Peer-Review Process,” Learned Publishing 15 no. 4 (October 2002): 247-258.


Importance of the Peer Review Process

0102030405060708090

100Peer-reviewed

Refs' commentspublished

Referees identified

Public commentary oneprints

Post-publication publiccommentary

Ability to submitcomments

http://www.alpsp.org/pub5.ppt

What is Gray/Grey Literature ?Papers are often written to inform funding bodies

about the results of research projects, to support grant applications, to inform rapidly a specific scientific community, to present preliminary results at conferences or as dissertations.

Such material is disseminated quickly, often in limited numbers, before or without the formal publication process. Such documents are called non-conventional or grey literature.

http://www.kb.nl/infolev/eagle/what_is_gl.htm

The Value of Grey LiteratureGrey literature is really a type of informal

communication, which on a scale of formality, fits in somewhere between conversation and

normal publication. A formal publication may follow later but in many cases - contrary to the

common assumption - these papers may not been made publicly available at all.


Nevertheless, grey publications may contain comprehensive, concrete and up-to-date information on research findings, and investigations have shown, that even when grey documents are published officially at a

later stage, detailed information on techniques, methods, measured values and details of experiments are

frequently omitted.

For these details of importance for further research, the non-conventional literature is then the first and only

source of information.


Veterinary Medicine12 Major Veterinary Medicine JournalsOverall, 6.38% of cited literature was Gray/Grey Literature

The figures for individual journals ranged from about 2.5 % to 10% gray/grey literature

Research journals cited a higher percentage of Gray/Grey Literature than did Clinical titles

William H. Weise and Nancy Pelzer, “Bibliometric Study of Grey Literature in Core Veterinary Medicine Journals,” Journal of the

Medical Library Association 91 no. 4 (October 2003): In press.

Indexing and Abstracting Services

SIGLE: System for Information on Grey Literature

National Technical Information Service (NTIS)

PsychINFO (Psychological Abstracts)

SIGLESystem for Information on Grey

LiteratureGrey literature documents covered

by SIGLE are technical or research reports, preprints, committee reports, working papers, dissertations, conference papers, discussion and policy papers, government reports, market surveys, etc.

http://www.kb.nl/infolev/eagle/frames.htm

SIGLESystem for Information on Grey

LiteratureNo. of Records | Category

4,158 | Aeronautics

17,044 | Agriculture, plant & veterinary sciences

17,668 | Environmental pollution, protection & control

256,657 | Humanities, psychology & social sciences

81,269 | Biological & medical sciences

25,089 | Chemistry

http://www.kb.nl/infolev/eagle/frames.htm

NTIS (National Technical Information Service)

NTIS Database provides bibliographic data and abstracts of unclassified and publicly

available information from research reports, journal articles, data files, computer programs and audio visual products, from U.S. and non-

U.S. governmental, organizational, and commercial sources

The NTIS Database produced by the National Technical Information Service, is the preeminent resource for

accessing the latest U.S. government-sponsored research and worldwide scientific, technical, engineering, and

business-related information.

http://www.csa2.com/csa/factsheets/ntis.shtml

Subject CoverageAdministration and Management

Aeronautics & Aerodynamics

Agriculture Behavior & Society

Business Chemistry Communications Computer Science

Education Energy Engineering Environmental Sciences

Health Care International Trade

Library & Information Science

Materials Sciences

Mathematical Sciences

Natural Resources & Earth Sciences

Nuclear Science Physics

Regulations Technology Tele-communications

Transportation

PsycINFOPsycINFO provides access to international literature in

psychology and related disciplines. Unrivaled in its depth of psychological coverage and respected worldwide for its high

quality, the database is enriched with literature from an array of disciplines related to psychology such as psychiatry, education, business, medicine, nursing, pharmacology, law, linguistics, and

social work.

http://www.csa2.com/csa/factsheets/psycinfo.shtml

PsycINFO includes psychological research and its applications; the database is of prime relevance to many industries and research

establishments worldwide. The sources include over 1,400 professional journals, chapters, books, reports, theses and

dissertations, published internationally.

Subject CoverageApplied Psychology

Communication Systems

Developmental Psychology

Educational Psychology

Experimental

Psychology

Personality Psychological and physical disorders

Professional personnel issues

Physiological Psychology and

Neuroscience

Psychometrics

And Statistics

Social Psychology

Treatment and Prevention

Database Coverage Size

SIGLE 1976 –

Present

781,410 records(November 2002)

NTIS 1964 - Present 2,168,400

records(October 2001)

PsycINFO 1872-

Present

1,870,180

records(September 2002)

DATABASE COVERAGE AND SIZE

http://www.neci.nec.com/~lawrence/papers/online-nature01/

Conference papers are typical gray/grey literature!

EPrints are Gray/Grey Literature

Daniela Luzi (1998) “E-Print Archives: a New Communication Pattern for Grey Literature,”

Interlending & Document Supply 26 no. 3 (1998): 130-139.

Gray/Grey Literature“It’s good enough,

it’s smart enough,

and

doggone it, people use it!”

With apologies to Stuart Smalley

http://www.openarchives.org/Register/BrowseSites.pl

http://software.eprints.org/

June 2003

http://eprints.anu.edu.au/

http://caltechcstr.library.caltech.edu/

http://eprints.lub.lu.se/

http://ndltdpapers.dlib.vt.edu:9090/

NETWORKED DIGITAL LIBRARY OF THESES AND DISSERTATIONS

http://dspace.org/index.html

http://rocky.dlib.vt.edu/~etdunion/cgi-bin/browse.pl

http://www.ncstrl.org/

Quality Control in Scholarly Publishing on the Web

"Most of the high quality materials on the Web are not peer-reviewed and much

of the peer-reviewed literature is of dubious quality.”

William Y. Arms, "What Are the Alternatives to Peer Review? Quality Control in Scholarly Publishing On The Web."

Journal of Electronic Publishing, 8 no. 1 (August 2002).

http://www.press.umich.edu/jep/08-01/arms.html

http://www.update-software.com/Cochrane/MR000016.pdf

Cochrane Methodology ReviewDespite its widespread use and costs, little hard evidence exists that peer review improves the quality of published biomedical research.

There had never even been any consensus on its aims and that it would be more appropriate to refer to it as ‘competitive

review’.Caroline White, “Little Evidence for Effectiveness of Scientific Peer Review,”

BMJ 326 (February 1, 2003): 241http://bmj.com/cgi/reprint/326/7383/241/a.pdf

Cochrane Methodology ReviewOn the basis of the current evidence, ‘the practice of peer review is based on faith in its effects, rather than on facts,' state the authors, who call for large, government funded research programmes to test the effectiveness of the [classic peer review] system and investigate possible alternatives.

Caroline White, “Little Evidence for Effectiveness of Scientific Peer Review,”BMJ 326 (February 1, 2003): 241

http://bmj.com/cgi/reprint/326/7383/241/a.pdf

Cochrane Methodology ReviewThe use of peer-review is usually assumed to raise the quality of the end-product (i.e. the journal or scientific meeting) and to provide a mechanism for rational, fair and objective decision-making. However, these assumptions have rarely been tested.

Tom O. Jefferson, Phil Alderson, Frank Davidoff, and Elizabeth Wager, Editorial Peer-review for Improving the Quality of Reports of

Biomedical Studies. (Middle Way, Oxford:Update Software Ltd, 2003).


Cochrane Methodology ReviewThe available research has not clearly identified or assessed the impact of peer-review on the more important outcomes (importance, usefulness, relevance, and quality of published reports)

… [G]iven the widespread use of peer-review and its importance, it is surprising that so little is known of its effects

Tom O. Jefferson, Phil Alderson,Frank Davidoff, and Elizabeth Wager, Editorial Peer-review for Improving the Quality of Reports of

Biomedical Studies. (Middle Way, Oxford:Update Software Ltd, 2003).


http://www.biomedcentral.com/news/20030203/04/

Royal Society

UNCITEDNESS

David P. Hamilton, "Publishing by and for? -- the numbers,” Science (New Series) 250 (4986) (December 7 1990): 1331-1332.http://www.garfield.library.upenn.edu/papers/hamilton1.html

David P. Hamilton,“Research Papers: Who’s Uncited Now?,” Science (New Series) 251 (4989) (January 4, 1991): 25

http://garfield.library.upenn.edu/papers/hamilton2.html

David P. Hamilton,“Research Papers: Who’s Uncited Now?,” Science (New Series) 251 (4989) (January 4, 1991): 25

http://garfield.library.upenn.edu/papers/hamilton2.html

REJECTED CITATION CLASSICSNOBEL PRIZE RESEARCH

Severo Ochoa

Polynucleotide phosphorylase

Hans Krebs

Citric acid cycle

Rosalind Yalow

Radioimmunoassay

Harmut Michel

Photosynthetic processesJuan Miguel Campanario, “Commentary: On Influential Books and Journal Articles

Initially Rejected Because of Negative Referees’ Evaluations, Science Communication 16 no. 3 (March 1995): 306-325

PEER REVIEW: PURPOSESPeer review helps to ensure that published research is:

Important ? Original ?

Timely ? Technically-reliable ?

Internally consistent ? Well-presented ?

Benefited from guidance by experts ?


FILTERING (1)UpStream / DownStream

“Researchers look at … [certain types] of electronic publications because, despite being tentative, may be relevant to their

work. Researchers are expected to do their own ‘downstream-filtering’ of relevant

information, which in the electronic world can be facilitated by providing meta-

information.”

FILTERING (2)UpStream / DownStream

Some have expressed the concern that having non-peer reviewed documents with peer-

reviewed documents on the same server would ‘contaminate’ the latter and compromise its

quality: Readers could have trouble in distinguishing

the different sections ‘Making non-peer-reviewed as well as peer-

reviewed material will confuse both scientists and the public … .’

FILTERING (3) UpStream / DownStream

“‘However, this perhaps belittles the ability of scientists to recognize different levels of evidence and to be able to interpret [quality] labels that could make clear that certain materials is non-peer-reviewed content’ after all this is

the age of transparency rather than paternalism … .”

Gunther Essenbach, “The Impact of Preprint Servers and Electronic Publishing on Biomedical Research,”

Current Opinion in Immunology 12 no. 5 (October 2000): 499-503.

http://yi.com/home/EysenbachGunther/scans/Eysenbach2000e_CurrOpImmunol_preprint_servers.pdf

INVISIBLE HAND OF CLASSICAL PEER REVIEW

“The refereed journal literature needs to be freed from both paper and its costs, but not from peer review, whose ‘invisible hand’ is what maintains its quality.”

Stevan Harnad

http://www.presidentmoron.com



“Human nature being what it is, it cannot be altogether relied upon to police itself. Individual exceptions there may be, but to treat them as the

rule would be to underestimate the degree to which our potential unruliness is vetted by

collective constraints, implemented formally.”

Stevan Harnad, “The Invisible Hand of Peer Review,” Exploit Interactive no. 5 (April 2000).

http://www.exploit-lib.org/issue5/peer-review/


“The system is not perfect, but it is what has vouchsafed us our refereed journal literature to

date, such as it is, and so far no one has demonstrated any viable alternative to having experts judge the work of their peers, let alone

one that is at least as effective in maintaining the quality of the literature as the present imperfect

one is.”


“Remove that invisible constraint -- let the authors be answerable to no one but the general users of the

Archive [arXiv.org] (or even its self-appointed "commentators") -- and watch human nature take its

natural course, standards eroding as the Archive devolves toward the canonical state of unconstrained

postings: the free-for-all chat-groups of Usenet … , that Global Graffiti Board for Trivial Pursuit -- until

someone re-invents peer review and quality control.”

Stevan Harnad, “The Invisible Hand of Peer Review,” Exploit Interactive no. 5 (April 2000).

http://www.exploit-lib.org/issue5/peer-review/

INVISIBLE HANDS

INVISIBLE HANDS

Personal reputation

Institutional reputation

Pride

Self-respect

Professional respect

Peer pressure

‘Critical Peer Response’

Invisible College

Self-Archiving-Process-Itself

Open access

Common Sense

Self-correcting dynamics

RECOMMENDATIONSWorkshop on the Open Archives Initiative (OAI)

and Peer Review Journals in Europe, CERN, Geneva Switzerland,March 22-24, 2001

“It was [also] generally believed that the electronic environment allows for novel

approaches to accord a stamp of quality to scholarly works.”

Alison Buckholtz, Raf Dekeyser, Melissa Hagemann, Thomas Krichel, and Herbert Van de Sompel, “Open Access: Restoring Scientific Communication to Its Rightful

Owners,” European Science Foundation Policy Briefing 21 (April 2003): 1-8.


Examples of new metrics that can be extracted from a fully electronic communication system are:

Usage counts of a work

Automatically extracted citation information with a scope beyond the ISI- core journals

Amount of discussion generated by a paper submitted in a system with open peer review and peer comment

Etc.Alison Buckholtz, Raf Dekeyser, Melissa Hagemann, Thomas Krichel, and Herbert Van de Sompel, “Open Access: Restoring Scientific Communication to Its Rightful

Owners,” European Science Foundation Policy Briefing 21 (April 2003): 1-8.


Scientific Publishing as RhetoricThe problems with peer review become evident once the fact that science has a rhetorical element is accepted. On the one hand, the traditional mode of peer review obscures the problems of reference and the rhetorical dimension of science. The rhetorical process which is at the heart of science and peer review conveniently disappears with the final publication of the manuscript. In its place is an ideal typical representation (the scientific paper) of the realist assumptions about empirical reference. All the academic world sees is a polished manuscript where the personal involvement of the researcher and reviewers has been systematically eliminated.

Mike Sosteric, “Interactive Peer Review: A Research Note,” Electronic Journal of Sociology 2 no. 1 (1996).

http://socserv.socsci.mcmaster.ca/EJS/vol002.001/SostericNote.vol002.001.html

‘IDEAL SPEECH SITUATION’

A theoretical construct that describes the ideal type of interpersonal interaction that should exist in a rhetorical situation.

Jürgen Habermas

IDEAL SPEECH SITUATIONthe ideal speech situation permits each interlocutor an

equal opportunity to initiate speech;there is mutual understanding between interlocutors;there is space for clarification; all interlocutors are equally free to use of any speech

act; there is equal power over the exchange.

Applied in the context of peer, the Ideal Speech Situation ‘would permit unimpeded authorial initiative, endless rounds of give and take, [and] unchecked openness among authors, editors, and referees.’

Mike Sosteric, “ Interactive Peer Review: A Research Note,” Electronic Journal of Sociology 2 no. 1 (1996).

http://socserv.socsci.mcmaster.ca/EJS/vol002.001/SostericNote.vol002.001.html

CONTINUAContinuum of PUBLICATION

(‘Scholarly Skywriting’)

WEAK

MEDIUM

Continuum of REVIEW (‘Scholarly Skyreading’)

STRONGSTRONG

OPEN ACCESS and

OPEN RETRIEVAL without

OPEN USEIncongruent

Contradictory Ironic

Paradoxical

Cognitively Dissonant

Three-Legged Stool

ACCESS -- RETRIEVAL -- USE

ACCESS

OPEN ACCESS INITIATIVES

RETRIEVAL

OPEN ARCHIVES INITIATIVE FOR METADATA HARVESTING

USE

OPEN SCHOLARSHIP

ACCESS -- RETRIEVAL -- USEACCESS -- RETRIEVAL -- USE

Lots of Alternative ModelsProvide Sensible Solutions

Four-Legged Stool

ACCESS -- RETRIEVAL -- USE -- NAVIGATION

INFORMATION OVERLOAD

OAIster

A search engine for freely available, difficult-to-access, academically-oriented digital resources that are OAI -compliant

http://oaister.umdl.umich.edu/

University of Michigan Digital Library Production Service

institutional repositories

departmental repositories

e-Journal collections

technical reports

dissertations and theses

discipline eprint collections

working papers

Internet resources

audio

video

images

Cognitive Psychology

Open Archives Initiative Protocol for Metadata Harvesting

(OAI-PMH)

METADATA ELEMENT for QUALITY

ADD

“Within the framework of OAI,

there is a need for a new protocol for certification.

There was strong support

for the extension of the usage of the

OAI protocol beyond discovery-related

metadata. Given the focus of the [1st OAI]

workshop on peer review, concrete

actions were suggested to address the

exchange of certification-related

metadata using the OAI protocol in a

trusted environment.”


Alison Buckholtz, Raf Dekeyser, Melissa Hagemann, Thomas Krichel, and Herbert Van de Sompel, “Open Access: Restoring Scientific Communication to Its Rightful Owners,” European

Science Foundation Policy Briefing 21 (April 2003): 1-8.

QUALITY METADATA (1)<oai-quality>


<process>

peer review

</process>

<organization>

CERN

</organization>

<policies>


</policies>

</oai-quality>

William Y. Arms, “Quality Control in Scholarly Publishing On The Web. What Are the Alternatives to Peer Review?” PowerPoint presentation given at Workshop on the Open Archives

Initiatives (OAI) and Peer Review Journals in Europe, March 22-24, 2003, CERN, Geneva, Switzerlandhttp://www.cs.cornell.edu/wya/papers/CERN-2001.ppt

QUALITY METADATA (2)<oai-quality>


<rating>

*****

</rating>

<organization>

42

</organization>

<policies>


</policies>

</oai-quality>

CERTIFICATION SERVICESFaculty of 1,000,000

CERTICATION SERVICESNew roles for Indexing and Abstracting Services

Expanded Role for Learned and Professional Societies

Establishment of Formal/Commercial Reviewing Services

PEER REVIEW: STRENGTHSThe underlying strength of peer review is“…the concerted effort by large numbers of researchers and scholars who work to assure that valid and valuable works are published and conversely to assure that invalid or non-valuable works are not published … .”

Anne C. Weller, Editorial Peer Review: Its Strengths and Weaknesses. (Medford, NJ: Information Today, 2001).

Faculty of 1000 / BioMed CentralBioMed Central (biomedcentral.com) publishes Faculty of 1000 (F1000), the leading literature evaluation service and “new online research tool that highlights the most interesting papers in biology, based on the recommendations of over 1000 leading scientists.” F1000 is managed “by scientists for scientists” …. [and] provide[s] a rapidly updated consensus map of the important papers and trends across biology.”

www.facultyof1000.com

Faculty of 1000 / BioMed CentralAmong its many benefits, F1000:systematically organizes and evaluates the mass of information within scientific literature; provides scientists with a continuously updated insider's guide to the most important papers within any given field of research;highlights papers on the basis of their scientific merit rather than the journal in which they appear;offers the researcher a consensus of recommendations from well over 1000 leading scientists; and,offers an immediate rating of individual papers by the authors' peers, and an important complement to the indirect assessment provided by the journal impact factor.

Faculty of 1000 / BioMed CentralWithin the F1000, the entire field of biology

is divided into 16 subject areas (‘Faculties’)(e.g., ‘Biochemistry,’, Cell Biology, ‘Microbiology’). Each ‘Faculty’ is subdivided into three (3) to twelve (12) ‘Sections,’ (e.g., Biochemistry: Biocatalysis, Molecular evolution, Protein folding), with each section comprised of between 10 to 50 faculty members. F1000 seeks to invite the best internationally known scientists in each represented field and to involve both experienced and younger investigators.

“Peer review is a quality-control and certification (QC/C) filter necessitated by the vast scale of learned research

today. Without it, no one would know where to start reading in the welter of new work reported every day, nor

what was worth reading, and believing, and trying to build one’s own further research upon.”

Stevan Harnad, “Free at Last: The Future of Peer-Reviewed Journals,”

D-Lib Magazine 5 no. 12 (December 1999)

http://www.dlib.org/dlib/december99/12contents.html

SEIZE THE E!

Embrace the potential of the digital environment to facilitate access, retrieval,

use, and navigation of electronic scholarship.

OPEN NAVIGATION

New Age Navigation:Innovative Interfaces for

Electronic Journals

Gerry McKiernan

The Serials LibrarianFall 2003

SUMMARY. While it is typical for electronic journals to offer conventional search features similar to those

provided by electronic databases, a select number of e-journals have also made available higher-level access options as well. In this article, we review several novel

technologies and implementations that creatively exploit the inherent potential of the digital environment to

further facilitate use of e-collections.

Gerry McKiernan, “New Age Navigation: Innovative Interfaces for Electronic Journals,”

The Serials Librarian, Fall 2003.

http://www.coleonline.us/serialslibrarian/

http://www.highwire.org

Topic Map

Topic Map

Topic Map

Topic Map

Topic Map

Secondary Screen

Topic Map

Topic Map

http://www.inxight.com

TopicMap is based on the Hyperbolic Tree SDK for Java, licensed from Inxight

Software, Inc., a spin-off company from Xerox PARC, and leading provider of

Unstructured Data Management solutions for accessing, analyzing and delivering

information.

Semio

Automated categorization software technologyhttp://www.entrieva.com/

Kohonen Self-Organizing Maps

http://websom.hut.fi/websom/milliondemo/html/root.html


http://websom.hut.fi/websom/milliondemo/html/1_cx5.html


http://websom.hut.fi/websom/milliondemo/html/2_gx10.html


http://websom.hut.fi/websom/milliondemo/getnd.cgi?32323


http://websom.hut.fi/websom/cgi-bin/getfile.cgi/comp.ai/39340

http://www.springer.de/books/toc/3540679219-c.pdf

“There are some excuses, but at the bottom it will be seen to be the

sluggishness of human nature and its superstitious cleavage to old habits.”

Stevan Harnad, “Free at Last: The Future of Peer-Reviewed Journals,”

D-Lib Magazine 5 no. 12 (December 1999)

http://www.dlib.org/dlib/december99/12contents.html

“… [In] the digital world, the evaluation process stands ready to be reinvented in a clear, rational

way by the relevant research communities themselves.”

Jean-Claude Guédon,

In Oldenburg’s Long Shadow: Librarians, Research Scientists, Publishers,

and the Control of Scientific Publishing.

(Washington, D.C.: Association of Research Libraries, •2001), 54.

http://www.arl.org/arl/proceedings/138/guedon.html

</ENDQUOTE>

“The Medium is the Message …

And

the Method.”

With apologies to Marshall McLuhen

THANK YOU FOR YOUR ATTENTION!

Alternative Peer Review:Quality Management

for 21st Century Scholarship

Gerry McKiernanScience and Technology Librarian

and Bibliographer

Iowa State University Library

Ames IA

[email protected]

OPEN MIND

REVISED VERSION 1.01

July 27, 2003

11:30 AM

Documents

Alternative Peer Review : Quality Management for 21 st Century Scholarship