The Refereeing Moduleof the SPMS
FEL2005: 22-26 August
Heinz-Dieter Nuhn – Scientific EditorBeck Reitmeyer – Conference Editor
Referee = Reviewer = ExpertRefereeing = Reviewing
History
• Some JACoW conference series use a refereeing process, like ICALEPCS, but it was not included in the SPMS
• For FEL, there has always been a refereeing process for the proceedings
• Through 2003, published through Elsevier• Joined JACoW in 2004; refereeing added
to SPMS initially by Ivan, but it was a separate instance
Adding Refereeing Module to the SPMS: Proposed Workflow
• Workflow affected by SLAC Computer Security; worked out OK anyway for editing
• Referees for this conference:– Assigned 4-8 papers– Given 6 weeks to complete
refereeing
Proposal for JACoW Editing with Refereeing
Editor RefereeAuthor
Submit paper through SPMS
system
Check paper for minimal
requirements
PS file distills properly?Template used? Page length
requirement met?
Return paper to author
No
Make correctionsSubmit paper to
referee for review
Yes
Review paper for scientific content
Does paper merit publication? Could the paper be improved with
changes?
Declined
Acceptable
Declined, with revisions #
RedOrange
Green
Perform “JACoW editing”
Does paper meet JACoW requirements?
Accepted
Changes acceptable?
Green
Yes
No
Correct source file
Return paper to author
Yellow
Red
# Declined with revisions: Major revisions recommended. If author does not choose to make them, paper will be declined.
Prior to deadline?
Yes
Declined
No
Severity of changes?
Minor
Major
How It Works: BeforeStep 1: System Parameter Setup
• Auto-assign referees• Text for e-mail notice for paper
available• Text for e-mail notice for
referee assigned• Instructions• Notify admin on decline (red)• Notify admin on revisions
(orange)• Notify PC on decline (red)• Notify PC on revisions
(orange)• Refereed conference• Referees per paper
How It Works: BeforeStep 2: Processing Code Setup
• (May be different already: separate screen for referee and editing codes)
• Add refereeing codes in at the same place you add in Editing Codes
ADDED LATER:• A “no paper” code and an “in
review” codeUNCLEAR:• Which takes precedence –
editing or refereeing code?
How It Works: BeforeStep 3: Assigning Referees
• Location: Maintenance Tables > Classifications > Main Classifications > Experts (at the very end of the line)
• Each classification has a set of referees assigned to it
• That referee within each classification can be assigned a paper
• Reports > Referee > Unassigned identifies papers that were missed
How It Works: DuringStep 4: Editor Checks
• Theory: Just a quick check, make a PDF• Reality: Full edit (OK – use the editors when you have them!)
• Kept future editing code and notes to ourselves and authors in the “Editor’s Notes” field, let author know that paper was being refereed in the “Comments to Author” field.
• Did not assign an Editing code during this first pass!• When the editor uploaded a PDF, e-mail sent to the referee.
PROBLEM:• Author could still upload a paper; adding in “In Review” code
would have stopped this, but again, which code takes precedence
How It Works: DuringStep 5: Refereeing
• Referees used three codes: Referee Green: Good paper, accepted for the proceedings Referee Orange: OK paper, needs some work before it can be accepted
for the proceedings Referee Red: Rejected for the proceedings based on scientific content
NOTE:• We expected mostly greens, got about half green, half oranges.• This meant that a lot (>70) of the papers were going to be rewritten
and resubmitted, most likely after the conference.PROBLEM (fixed):• System wasn’t sending out notices of Referee Orange dot to
authors; some found it on their own, some didn’t.• A lot of people didn’t know what the Paper Dot Status Board was
for.
How It Works: DuringStep 5: Refereeing (2)
• Regular login• Extra “Referee” link available
• New screen listed all assigned papers
• If a paper was available for refereeing (signified by a PDF having been uploaded by an editor), then the Paper ID showed as a link
How It Works: DuringStep 5: Refereeing (3)
• Referees were asked to use the color codes and provide comments to the authors
PROBLEM:• Unclear what made the
paper orange (content or style); needed clearer instructions
• 4000 characters was not enough for some
(Aside) Refereeing Timeline
• Most of the referees did not referee their papers during the conference
• Authors theoretically had 2 weeks to respond; almost all did, but not necessarily within 2 weeks
• ~98% of papers with an orange dot were resubmitted
Paper Processing - Refereeing
020406080
100120140160180
8/25
/200
5
9/1/
2005
9/8/
2005
9/15
/200
5
9/22
/200
5
9/29
/200
5
10/6
/200
5
10/1
3/20
05
10/2
0/20
05
10/2
7/20
05
11/3
/200
5
Date
Nu
mb
er o
f P
aper
s
In ReviewGreenOrangeRed
How It Works: AfterStep 6: Reprocessing
• Resubmitted papers were processed as before• Second review done by the Scientific Editor, not the
original referee• Second referee notice e-mail was just sent to the
administrator e-mail (a mailing list)
How It Works: AfterStep 7: Finishing
• When a paper passed the refereeing process (that is, it got a Referee Green dot), it came back to the editors for final processing
• Basically, this meant copying the notes from the “Editor’s Notes” field to the “Comments to Author” field and selecting the correct editing dot
How It Works: AfterStep 7: Finishing (2)
• Because most of the papers marked Editing Yellow were notified as such after the conference, we stopped notifying authors since they either didn’t respond or responded with many corrections
• Papers were proofed against the author copy or carefully on its own
• Usually, this final proofing has occurred as part of the final QA
Actual Process for JACoW Editing with Refereeing
RefereeEditorAuthor
No (Red) Yes
RedOrange
Green
Green
Yes
No
Yellow
Minor
Major
Changes acceptable?
Accepted
Declined
Confirm “JACoW editing” dot
Submit paper to referee for review
# Declined with revisions: Revisions recommended.
Review paper for scientific content
Declined, with revisions #
Submit paper through SPMS
system
Does paper merit publication? Could the paper be improved with
changes?
Return paper to author
Perform “JACoW editing”
Does paper meet JACoW requirements?
Correct source file
Make corrections
Acceptable
Severity of changes?
During or After the Conference?
During
Carefully proofread paper
After
Which version?
Assigned Referee Scientific Editor
1 ≥2
Actual Workflow1. “JACoW editing” took
place before refereeing; dotting took place after for Editing Green and Editing Yellow dots
2. After the first version of the paper, it went to the Scientific Editor, not the named referee
3. Editing Yellow papers after the conference were not sent back to the author, but checked by the editors
1
2
3
Next Time: Clearly Needed• Add the additional codes (at least “In
Review”) first• Allow searching on the refereeing
code• Put refereeing statistics on main
Statistics page or on the refereeing-only Master Status page? Definitely needs tallies somewhere.
• (may be done) Separate out the refereeing codes from the editing codes– Determine best place for “in review”
dot – with editor or referee?– Determine which code takes
precedence for author uploads?• (done) Referees could see “owner”
of the comments on their submissions
Next Time: Needed, but Confusing
• Modify the main Statistics report page to include refereeing – numbers of where papers stand is very hard to figure out
• Really hard to get a handle on what’s going on from an administrator perspective – papers come back to editors, and after the conference, they’re gone and not checking. This probably only happens with refereeing since there are steps that need to be taken after the paper comes back, and that mostly happens after the conference.
• Timing issue of editing vs. refereeing? Can/should this be a system parameter, or better handled “outside” the system like we did here? More process than programming. How does it affect precedence of referee vs. editing codes?
Suggested System Parametersfor Refereeing
• Refereed conference (Y/N)• Auto-assign referees (Y/N)• Text for e-mail notice for paper available (Text)• Text for e-mail notice for referee assigned (Text)• Link for instructions (Link, not full text)• Notify admin on decline (Y/N)• Notify admin on revisions (Y/N)• Notify admin on accepted (Y/N)• Referees per paper (# -- incomplete; both at the same
time?)• Second review (who? Number of reviews?)• Notify referee if referee dot is already green? (Y/N)
Not needed ifadmin is cc’d?
Next Time: Would Be Nice
• Determine statistics for refereeing– How many papers were initially orange?– How many papers are orange now?– How many papers were resubmitted more than once?
• Increase the number of characters allowed for the referees to >4000
• Put processing code in the subject line of the e-mails sent to authors/administrators
• Change “Edit Complete” to just “Editor”