The Refereeing Moduleof the SPMS

FEL2005: 22-26 August

Heinz-Dieter Nuhn – Scientific EditorBeck Reitmeyer – Conference Editor

Referee = Reviewer = ExpertRefereeing = Reviewing

History

• Some JACoW conference series use a refereeing process, like ICALEPCS, but it was not included in the SPMS

• For FEL, there has always been a refereeing process for the proceedings

• Through 2003, published through Elsevier• Joined JACoW in 2004; refereeing added

to SPMS initially by Ivan, but it was a separate instance

Adding Refereeing Module to the SPMS: Proposed Workflow

• Workflow affected by SLAC Computer Security; worked out OK anyway for editing

• Referees for this conference:– Assigned 4-8 papers– Given 6 weeks to complete

refereeing

Proposal for JACoW Editing with Refereeing

Editor RefereeAuthor

Submit paper through SPMS

system

Check paper for minimal

requirements

PS file distills properly?Template used? Page length

requirement met?

Return paper to author

No

Make correctionsSubmit paper to

referee for review

Yes

Review paper for scientific content

Does paper merit publication? Could the paper be improved with

changes?

Declined

Acceptable

Declined, with revisions #

RedOrange

Green

Perform “JACoW editing”

Does paper meet JACoW requirements?

Accepted

Changes acceptable?

Green

Yes

No

Correct source file

Return paper to author

Yellow

Red

# Declined with revisions: Major revisions recommended. If author does not choose to make them, paper will be declined.

Prior to deadline?

Yes

Declined

No

Severity of changes?

Minor

Major

How It Works: BeforeStep 1: System Parameter Setup

• Auto-assign referees• Text for e-mail notice for paper

available• Text for e-mail notice for

referee assigned• Instructions• Notify admin on decline (red)• Notify admin on revisions

(orange)• Notify PC on decline (red)• Notify PC on revisions

(orange)• Refereed conference• Referees per paper

How It Works: BeforeStep 2: Processing Code Setup

• (May be different already: separate screen for referee and editing codes)

• Add refereeing codes in at the same place you add in Editing Codes

ADDED LATER:• A “no paper” code and an “in

review” codeUNCLEAR:• Which takes precedence –

editing or refereeing code?

How It Works: BeforeStep 3: Assigning Referees

• Location: Maintenance Tables > Classifications > Main Classifications > Experts (at the very end of the line)

• Each classification has a set of referees assigned to it

• That referee within each classification can be assigned a paper

• Reports > Referee > Unassigned identifies papers that were missed

How It Works: DuringStep 4: Editor Checks

• Theory: Just a quick check, make a PDF• Reality: Full edit (OK – use the editors when you have them!)

• Kept future editing code and notes to ourselves and authors in the “Editor’s Notes” field, let author know that paper was being refereed in the “Comments to Author” field.

• Did not assign an Editing code during this first pass!• When the editor uploaded a PDF, e-mail sent to the referee.

PROBLEM:• Author could still upload a paper; adding in “In Review” code

would have stopped this, but again, which code takes precedence

How It Works: DuringStep 5: Refereeing

• Referees used three codes: Referee Green: Good paper, accepted for the proceedings Referee Orange: OK paper, needs some work before it can be accepted

for the proceedings Referee Red: Rejected for the proceedings based on scientific content

NOTE:• We expected mostly greens, got about half green, half oranges.• This meant that a lot (>70) of the papers were going to be rewritten

and resubmitted, most likely after the conference.PROBLEM (fixed):• System wasn’t sending out notices of Referee Orange dot to

authors; some found it on their own, some didn’t.• A lot of people didn’t know what the Paper Dot Status Board was

for.

How It Works: DuringStep 5: Refereeing (2)

• Regular login• Extra “Referee” link available

• New screen listed all assigned papers

• If a paper was available for refereeing (signified by a PDF having been uploaded by an editor), then the Paper ID showed as a link

How It Works: DuringStep 5: Refereeing (3)

• Referees were asked to use the color codes and provide comments to the authors

PROBLEM:• Unclear what made the

paper orange (content or style); needed clearer instructions

• 4000 characters was not enough for some

(Aside) Refereeing Timeline

• Most of the referees did not referee their papers during the conference

• Authors theoretically had 2 weeks to respond; almost all did, but not necessarily within 2 weeks

• ~98% of papers with an orange dot were resubmitted

Paper Processing - Refereeing

020406080

100120140160180

8/25

/200

5

9/1/

2005

9/8/

2005

9/15

/200

5

9/22

/200

5

9/29

/200

5

10/6

/200

5

10/1

3/20

05

10/2

0/20

05

10/2

7/20

05

11/3

/200

5

Date

Nu

mb

er o

f P

aper

s

In ReviewGreenOrangeRed

How It Works: AfterStep 6: Reprocessing

• Resubmitted papers were processed as before• Second review done by the Scientific Editor, not the

original referee• Second referee notice e-mail was just sent to the

administrator e-mail (a mailing list)

How It Works: AfterStep 7: Finishing

• When a paper passed the refereeing process (that is, it got a Referee Green dot), it came back to the editors for final processing

• Basically, this meant copying the notes from the “Editor’s Notes” field to the “Comments to Author” field and selecting the correct editing dot

How It Works: AfterStep 7: Finishing (2)

• Because most of the papers marked Editing Yellow were notified as such after the conference, we stopped notifying authors since they either didn’t respond or responded with many corrections

• Papers were proofed against the author copy or carefully on its own

• Usually, this final proofing has occurred as part of the final QA

Actual Process for JACoW Editing with Refereeing

RefereeEditorAuthor

No (Red) Yes

RedOrange

Green

Green

Yes

No

Yellow

Minor

Major

Changes acceptable?

Accepted

Declined

Confirm “JACoW editing” dot

Submit paper to referee for review

# Declined with revisions: Revisions recommended.

Review paper for scientific content

Declined, with revisions #

Submit paper through SPMS

system

Does paper merit publication? Could the paper be improved with

changes?

Return paper to author

Perform “JACoW editing”

Does paper meet JACoW requirements?

Correct source file

Make corrections

Acceptable

Severity of changes?

During or After the Conference?

During

Carefully proofread paper

After

Which version?

Assigned Referee Scientific Editor

1 ≥2

Actual Workflow1. “JACoW editing” took

place before refereeing; dotting took place after for Editing Green and Editing Yellow dots

2. After the first version of the paper, it went to the Scientific Editor, not the named referee

3. Editing Yellow papers after the conference were not sent back to the author, but checked by the editors

1

2

3

Next Time: Clearly Needed• Add the additional codes (at least “In

Review”) first• Allow searching on the refereeing

code• Put refereeing statistics on main

Statistics page or on the refereeing-only Master Status page? Definitely needs tallies somewhere.

• (may be done) Separate out the refereeing codes from the editing codes– Determine best place for “in review”

dot – with editor or referee?– Determine which code takes

precedence for author uploads?• (done) Referees could see “owner”

of the comments on their submissions

Next Time: Needed, but Confusing

• Modify the main Statistics report page to include refereeing – numbers of where papers stand is very hard to figure out

• Really hard to get a handle on what’s going on from an administrator perspective – papers come back to editors, and after the conference, they’re gone and not checking. This probably only happens with refereeing since there are steps that need to be taken after the paper comes back, and that mostly happens after the conference.

• Timing issue of editing vs. refereeing? Can/should this be a system parameter, or better handled “outside” the system like we did here? More process than programming. How does it affect precedence of referee vs. editing codes?

Suggested System Parametersfor Refereeing

• Refereed conference (Y/N)• Auto-assign referees (Y/N)• Text for e-mail notice for paper available (Text)• Text for e-mail notice for referee assigned (Text)• Link for instructions (Link, not full text)• Notify admin on decline (Y/N)• Notify admin on revisions (Y/N)• Notify admin on accepted (Y/N)• Referees per paper (# -- incomplete; both at the same

time?)• Second review (who? Number of reviews?)• Notify referee if referee dot is already green? (Y/N)

Not needed ifadmin is cc’d?

Next Time: Would Be Nice

• Determine statistics for refereeing– How many papers were initially orange?– How many papers are orange now?– How many papers were resubmitted more than once?

• Increase the number of characters allowed for the referees to >4000

• Put processing code in the subject line of the e-mails sent to authors/administrators

• Change “Edit Complete” to just “Editor”